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Software Assurance Technology Center 


Develop and apply assurance technology for 
software products 

Primary task areas: 

• Software Metrics 

• Assurance Tools & Techniques 

• Guidebooks & Standards 

• Applied Research and Project Support 
Web page:http://satc.gsfc.nasa.gov 
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Facilities 



Restrooms 
Emergency exits 
Messages/phones 
Lunch/breaks 
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Targeted Audience 


Mix of project personnel and change agents 
with variable levels of experience development 
projects 

Prerequisites: 

• engineering experience (at least one year) 
Assumptions: 

* prior knowledge of risk or risk management 
unnecessary 
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Course Objectives 



Understand the concepts and principles of 
Continuous Risk Management and how to apply 
them 


Develop basic risk management skills for each 
component of Continuous Risk Management 

Be able to use key methods and tools ' 

Be able to tailor Continuous Risk Management 
to a project 
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Course Schedule 


One Day 


1 . Welcome 

7. Control 

2. Paradigm Overview 

8. Communicate & Document 

3. Identify 

9. Getting Started in Continuous 

4. Analyze 

Risk Management 

5. Plan 

10. Summary 

6. Track 

NASA SATC 
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Style of Course 

Interactive 

Lecture mixed with examples and discussion 
topics 

Exercises 

Case study - hypothetical but NASA-based 
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Course Materials 


Student notebook 

• Case study 

• List of Risks 


Continuous Risk Management Guidebook 
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Module 2 


Introduction 


1 NASA SATC 2' 1 


Overview 



What is risk? 

How is risk related to project management? 

Why do risk management? 

What is continuous risk management? 

Drivers for continuous risk management? 
Where is continuous risk management applied? 
When should risk management be done? 

Risk Management Plan 

Who does continuous risk management? 
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Definitions of Risk 
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Risk Management & 
Project Management 



Why Do Risk Management? 

•Early identification of potential problems 

•Increase chances of project success 

•Enable more efficient use of resources 

•Promote teamwork by involving personnel at all 
levels of the project 

•Information for tradeoffs based on priorities 
and quantified assessment 
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What is Continuous Risk Management? 


A management practice with processes, methods, 
and tools for managing risks in a project 

It provides a disciplined environment for proactive 
decision making to: 

• assess continually what could go wrong (risks) 

• determine which risks are important to deal with 

• implement strategies to deal with those risks 

• assure, measure effectiveness of the 
implemented strategies 
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Components of 

Continuous Risk Management - 1 

Identify 

• search for and locate risks before they 
become problems 

Analyze 

• convert risk data into useable information 

for determining priorities and making decisions 

Plan 

• translate risk information into planning 
decisions and mitigating actions (both present 
and future), and implement those actions 
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Components of 

Continuous Risk Management - 2 

Track 

* monitor risk indicators and mitigation actions 

Control 

* correct for deviations from the risk mitigation 
plans and decide on future actions 

Communicate & Document 

* provide information and feedback to the project 
on the risk activities, current risks, and 
emerging risks 
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Relationship Among Functions 


Throughout the project life cycle, risk components 
evolve 

• continuously 

• concurrently 

• iteratively 

/ / \ 

/ / y \ 

rA 

1 3 | Communicate 

\ ** Document / 
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Risk Management Data Flow 












rusit management Data * low 















Drivers for Continuous Risk Management 


•NASA NPG 7120.5A: NASA Program and Project 
Management Process and Requirements 

•NASA-SP-6105: NASA Systems Engineering Handbook 

•ISO 9001: Quality systems 

•OMB Circular A-1 1 : Planning, Budget & Acquisition 

•IEEE: PI 448 - EIA PN3764 (ISO/IEC 12207): Standard for 
Information Technology 

•DoD: Military Standard Handbook 338: Electronic & 
Reliability Design Handbook 

•DoD: Military Standard 499: Engineering Management 
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Where is Continuous Risk 
Management Applied? 






When Should Continuous Risk 
Management be Done? 



System 

Requirements 

Analysis 

System 

Design 


yka<° I Detailed 

Preliminary Pe8ifln 
Hardware Design 
equiremen 
Analysis 


Testing 


Fabrication 


t ^ 




Software \ 

Requirements 

Analysis Preliminary 

\ Design 


Document A 

j . y/j 


Code* 
I Debug 


Integration | 


Risk Management Plan 


Definition 

• documents the risk management practice 
(processes, methods, and tools) to be used for a 
specific project 

Contents 

• overview 

• project organization, roles, responsibilities 

• practice details (e.g., how are risks identified?) 

• risk management milestones (e.g., quarterly 
rebaselining) 

• risk information documentation (e.g., database) 


Guidebook pp. 451-455 






Relationship to Everyday Practice 


Learning 

Continuous Risk Management 
is similar to incorporating 
any new habit 
into your daily life. 
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Who Does Continuous Risk 
Management? 
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Module 3 


Identify 


NASA SATO 


Overview 


Activities overview 

Identification activities 

• capturing statements of risk 

* capturing the context of a risk 

Identification methods and tools 

• Examples 

‘ Brainstorming 

* Questionnaires and checklists 
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Identification Activities Overview 


Individual 

uncertainties 


Group/team 

uncertainties 


Identify 

• capture statement 
of risk 

• capture context of 

risk j 


Project 

data 


Statement of risk 


Context 


List of risks 


Recording Data on Risk 
Information Sheet 


-Risk information sheet 
-NASA Risk 
Management database 

Complete: 

• ID 

• Date Identified 

• Risk statement 

• Origin 

• Risk Context 
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Risk Information Sheet 
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Capturing Statements of Risk - 1 
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Elements of a Good Risk Statement 


Consider these questions when looking at a risk 
statement: 

• Is it clear and concise? 

• Will most project members understand it? 

• Is there a clear condition or source of concern? 

• If a consequence is provided, is it clear? 

• Is there only ONE condition followed by one (or 
more) consequence? 
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Example Risk Statements 

Good or bad risk statements? 

1 . Object Oriented Development ! 

2. The staff will need time and training to learn 
object oriented development 

3. This is the first time that the software staff will 
use OOD; the staff may have a lower-than- 
expected productivity rate and schedules may 
slip because of the associated learning curve. 


NASA SATC 


Rev 2. 1/99 






Ckse Study htnoduction 



IR-SIP Risk Statement Example #1 

Commercial parts are being selected for space 
flight applications, and their suitability to meet 
environmental conditions is unknown; these parts 
may fail to operate on-orbit within the environment 
window, leading to system level failures. Also, 
environmental testing of these parts can be 
expensive and cause schedule delays. 
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“2. A new high-speed fiber-optic data bus will 
be used so that high data transfer rates can 
be sustained.” 


Risk #2 : 

The high-speed fiber-optic data bus is untested 
technology; the bus may not perform as 
specified and high data transfer rates might 
not be sustained. 
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Capturing the Context of a Risk 


Purpose: 

• provide enough additional information about the 
risk to ensure that the original intent of the risk 
can be understood by other personnel, particularly 
after time has passed 

Description: 

* capture additional information regarding the 
circumstances, events, and interrelationships not 
described in the statement of risk 
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Context 


An effective context captures the what, when, 
where, how, and why of the risk by describing 
the circumstances, contributing factors, and 
related issues (background and additional 
information that are NOT in the risk statement). 
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Elements of Good Context? 


Consider these questions when looking at the context. 

• Can you identify which risk statement this context is 
associated with? 

• Is it clear what the source or cause of the risk is? 

• Is it clear what the impact might be? 

• Would you know who to assign the risk to for 
mitigation? Would they know what to do? 

• Would you be able to tell if the risk has gone away? 


| NASA SATC 3-15 

Rev 2, 

Example Context - 1 



Risk statement: 

This is the first time that the software staff will use 
OOD; the staff may have a lower-than-expected 
productivity rate and schedules may slip because 
of the associated learning curve. 

Good or bad context? 

* It’s a typical NASA project - new concepts 
without training. 
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Example Context - 2 


Risk statement: 

This is the first time that the software staff will use 
OOD; the staff may have a lower-than-expected 
productivity rate and schedules may slip because 
of the associated learning curve. 

Context: 

Object oriented development is a very different 
approach that requires special training. There will 
be a learning curve until the staff is up to speed. 
The time and resources must be built in for this or 
the schedule and budget will overrun. 
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Example Context - 3 



Risk Statement : Commercial parts are being selected for space 
flight applications and their suitability to meet environmental 
conditions is unknown; these parts may fail to operate on-orbit 
within the environment window, leading to system level failures. 
Also, environmental testing of these parts can be expensive and 
cause schedule delays. 

Context :Although commercial parts are more readily available and 
have lower prices than space qualified parts, they have not been 
subjected to space environment conditions or levels. In particular, 
radiation effects can cause these parts to fail since they were 
manufactured without radiation in mind. Radiation testing can be 
expensive, and if the selected parts fail to meet requirements, 
procurement of space qualified replacement parts have long 
procurement lead times. 
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Exercise: Writing Risk Statements 

Based on the material you have just read, working with your group, write 2-3 risk 
statements. When you are done chose one risk and write it on the board. 


Condition 


5 


Consequence 





Risk Statement Sample Solutions 


• This is the first time the IR Instrument Project manager is managing a 
project to go into space; Project may fail due to insufficient / poor 
management 

• There is a lack of a thorough hardware test program; mission failure 
due to environmental conditions not tested. 

• Project software schedule and resources were underestimated; 
Schedule slips, cost overruns, and a reduction in adequate testing 
time are likely results. 

• Science requirements have substantial TBDs; late completion of 
TBDs likely, with reduction in adequate testing time, possible science 
application software failure, incorrect science data being captured, 
hardware damage if incorrect safety limits were provided, extensive 
rework and substantial cost overruns, mission failure if problems not 
found before system is in operation. 
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Brainstorming 



Purpose: 

• group method for generating ideas 


Description: 

• participants verbally identify ideas as they 
think of them, thus providing the opportunity 
for participants to build upon or spring off of 
ideas presented by others 
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Software Development Risk 


Product 

Engineering 


Development 

Environment 


Program 

Constraints 


Element R«qulrM»nt.” sjSSStfcl? D ‘pS^ ,n *' • Environment R««ourc«. • • Ext.rn.ls 


Attribute se- frmm **»**•- 
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Goal/Question/Metric Paradigm GQM 

Mechanism for formalizing the characterization, 
planning, construction, analysis, learning and 
feedback tasks 

Three Steps: 

1. Generate a set of goals based upon the needs 
of the organization. 

2. Derive a set of questions. 

3. Develop a set of metrics which provide the 
information needed to answer the questions. 

(Solution to: How do we start?) 
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NASA Software Checklist 
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Mil Std 338 Design Checklist - Partial 


- Is the design simple? Minimum number of parts? 

- Are there adequate indicators to verify critical functions? 

- Are reliability requirements established for critical items? 

- Are standard high-reliability parts being used? 

- Have parts been selected to meet reliability requirements? 

- Are circuit safety margins ample? 

- Has provision been made for the use of electronic failure 
prediction techniques, including marginal testing? 

- Have normal modes of failure and magnitude of each mode for 
each item or critical part been identified? 

- Has redundancy been provided where needed to meet specified 
reliability? 

- Does the design account for early failure, useful life and wear out? 
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Identification Summary 








Risk Information Sheet 






Continuous Risk Management 


Case Study 

Risk information Sheet After Analysis 


ID 11 

Risk Information Sheet Identified: 

11/ 1/ 95 

Priority 10 

Statement 

It has recently been decided that the Infrared sensors will be 
developed in-house and how they will communicate and how sensor 
data will be processed will be based on assumptions until the detailed 
design is baselined; the accuracy and completeness of those 
assumptions will determine the magnitude of change in the IR-SIP 
Instrument Controller Cl and Infrared Sensing Unit Cl interface 
requirements - it could be minor or catastrophic. 

Probability M 

Impact H 

Timeframe N 

Origin Class Assigned 

K. Green Requirements to: 

Context The AA program is in the Systems Preliminary Design Phase and the IR-SIP project 


software is in the Software Specification Phase. 

• This is the first time these sensors will be used on a NASA mission. They will still be 
under design and definition during the IR-SIP Controller’s software specification through 
im plementation phases. Therefore, assumptions about the interface will have to be made in 
implementing the IR-SIP CSCI and if those assumptions are incorrect, then software 
rewrites will be necessary. We do have access to a reasonable set of assumptions and 
information from a contractor who has developed very similar sensors, but again, we don’t 
really feel 100% confident in those assumptions. 

• Problems were not anticipated in the current success-oriented schedule so there is no slack 
time if the impact of the changes is major. Schedule slips, cost overruns, and reduction in 
adequate testing time are all possible if the assumptions prove false. 

• System testing does not begin until very late in the development, so if problems are 

encountered there is usually no time to make changes in the hardware. Therefore, software 
must provide work-arounds for problems encountered. 

Mitigation Strategy 


Contingency Plan and Trigger 


Status 


Status Date 


Approval 


Closing Date 

/ / 


Closing Rationale 
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Identification Key Points 


[ Condition71 ~ H Consequence I 

A good risk statement Risk Statement 

• contains ONLY one condition 

• contains at least one consequence 

* is clear and concise 

Good context 

* provides further information not in the risk 
statement 

* ensures that the original intent of the risk can 
be understood by other personnel, even after 
time has passed 

• Communication is an integral part of risk 
identification. 


Identification Methods and Tools 


- Risk information sheet 
• Brainstorming 

- Periodic risk reporting 

- Voluntary Risk Reporting 

- Taxonomy-based questionnaire (TBQ) 

- Project metrics and Goal/Question/Metric* 

- NASA software risk checklist* 

- Mil-Std 338: Electronic & Reliability Design Handbook (HW)* 


* Not in Guidebook 
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Module 4 


Analyze 


NASA SATC 


Overview 


Analysis activities overview 

Analysis activities 

• evaluating attributes of risk 

• classifying risks 

• prioritizing risks 
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Analysis Activities Overview 












Evaluating Attributes of Risk 


Purpose: 

to gain a better understanding of the risk by 
determining the expected impact, probability, 
and timeframe of a risk 


Description - involves establishing values for: 
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Levels of Analysis 



Level 

Impact 

Probability 

Timeframe 

binary level 

significant 

likely 

near 


insignificant 

not likely 

far 

tri-level 

high 

high 

near 


moderate 

moderate 

mid 


low 

low 

far 

5-level 

very high 

very high 

imminent 


high 

high 

near 


moderate 

moderate 

mid 


low 

low 

far 


very low 

very low 

very far 

n-level 

n levels of 

n levels of 

n levels of 


impact 

probability 

timeframe 
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Example - 

Tri-Level Attribute Evaluation 


Each attribute has one of three values 

• Impact: catastrophic, critical, marginal 

• Probability: very likely, probable, improbable 

• Timeframe: near-term, mid-term, far-term 


Risk Exposure 


Impact 


Catastrophic 

Critical 

Marginal 


Probability 

Improbable^ 


Low 


Example: NASA Safety Impact Definitions 

Catastrophic 

Marginal 

• loss of entire 

• minor system 

system 

damage 

• loss of human life 

• minor injury (e.g., 

• permanent 

scratch) 

disability 


Critical 

Negligible 

• major system 

* no system damage 

damage 

• no injury (possibly 

• severe injury 

some aggravation) 

• temporary disability 
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Example - Impact Definitions 



Catastrophic 

Critical 

Marginal 

Schedule 

slip 

> 20% 

10-20% 

0-10% 

Cost 

overrun 

> 25% 

10-25% 

0 - 10% 

Failure 

System is 
lost 

Major 

function 

lost 

Data lost 
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Example Timeframe Definitions 

A risk is near-term if the project must take 
action or will be impacted by the risk in the next 
90 days. 

A risk is mid-term if the project must take 
action or will be impacted by the risk in the next 
90-180 days. 

A risk is far-term if the project need not take 
action or will not be impacted by the risk in the 
next 180 days. 
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Example Probability Definitions 


A risk is very likely if there is a >70% 
probability that it will occur. 

A risk is probable if there is a 30-70% 
probability that it will occur. 

A risk is improbable if there is a <30% 
probability that it will occur. 


NASA SATC 4.1 1 Rev 2. 1/&9 


Exercise 


Tri-Level Attribute Evaluation 
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Criteria and Attributes for IR-SIP 


Jerry Johnstone’s criteria for what’s currently important to the project: 

• must meet the schedule 

• can’t delete any of the technical or performance requirements 

• must keep to the budget (Jerry knows there’s a small amount of slack 
in the budget, but he doesn’t want the project personnel to know.) 


Attribute 

Value 

Description 

Probabibry 

Very Likely (H) 
Probable (M) 
Improbable (L) 

High chance otths nsk occunaig, thus becoming a problem >/U% 
Risk like this may turn into a problem once ii a while 30% < x < 70% 
Not much chance this will become a problem 0% < x < 30% 

Impact 

Catastrophic 

(H) 

Critical (M) 
Ma rg m a 1(L) 

toss of 1K-5IP; unrecoverable taiJure oi IK-SIP operations; major system 
damage to IR-SIP; schedule slip that causes vehicle launch date to be 
missed; cost overrun exceeding 50%ofphnncd costs, 

Minor system damage to IR-SIP with recoverable operational capacity; cost 
overrun cxceedng 10% (but less than 50%) ofplanncd costs. 

Miior system damage to IR-SIP. recoverable loss of IR-SIP operational 
capacity; rtemal schedule sljp that does not knpact vehicle bunch date; cost 
overrun of less than 10% of planned costs 

Imttrame 

Near-term (N) 
Mid-term (M) 
Far-term (F) 

Note: Refers to when action must be taken on the nsk m the next month 
1-2 months from now 
3 or more months from now 
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Continuous Risk Management 


Case Study 

Tri-level Attribute Evaluation 


Case Study Setting: It is October 20, 1995. The IR-SIP project is behind schedule in completing 
the Systems Requirements and Design. These are running in parallel. Both the IR-SIP Flight and 
the Mission Software have started requirements definition. The Science requirements are still 
incomplete and the AA Interface requirements are behind schedule. 


Key: Using the IR-SIP criteria description, evaluate each risk with respect to: 



Value 

Description 

Probability 

Very Likely (H) 
Probable (M) 
Improbable (L) 

High chance ot this risk occurring, thus becoming a problem >70% 
Risk like this may turn into a problem once in a while 30% < x < 70% 
Not much chance this will become a problem 0% < x < 30% 

Impact 

Catastrophic (H) 

Critical (M) 
Marginal(L) 

Loss of IR-SIP; unrecoverable failure ot IR-SlP operations; major 
system damage to IR-SIP; schedule slip that causes vehicle launch 
date to be missed; cost overrun exceeding 50% of planned costs. 

Minor system damage to IR-SIP with recoverable operational capacity; 
cost overrun exceeding 10% (but less than 50%) of planned costs 

Minor system damage to IR-SIP; recoverable loss of IR-SIP operational 
capacity; internal schedule slip that does not impact vehicle launch date; 
cost overrun of less than 10% of planned costs. 

Timeframe 

Near-term (N) ” 

Mid-term (M) 
Far-term (F) 

Note: Refers to when action must be taken on the risk. In the next 
month 

1-2 months from now 
3 or more months from now 
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HH 

Time- 

frame 


This is the first time that the software staff will use OOU; 
The staff may have a lower-than-expected productivity 
rate and schedules may slip because of the associated 
learning curve. 




— 2 

Commercial parts suitability for space applications is 
unknown; parts failure may lead to system failure and 
use of space grade parts may cause schedule delays 
since space qualified parts procurement have a 
procurement iead time of at least 1 8 months. 




3 

The high-speed fiber optic data bus is untested 
technology; the bus will not perform as specified and 
high data transfer rates will not be sustained. 




4 

First time the IK instrument Project manager is 
managing a project to go into space; Project may fail due 
to insufficient / poor management. 




wm 

Lack of a thorough hardware test program; mission 
failure due to environmental conditions not tested. 





Project software schedule and resources were 
underestimated; Schedule slips, cost overruns, and a 
reduction in adequate testing time are likely results. 
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Tri-Level Attribute Evaluation 


Setting: October 20th, the IR-S1P project is behind schedule in completing the Systems 

requirements and Design. These are running in parallel. Both the Flight and Mission 
segments have started requirements definition. The Science requirements are still 
incomplete and the AA interface requirements are behind schedule. 


is is the first time that the software staff will use OOD; 1 be staff may bave a 
I lo we r-t ha n-expected productivity rate and schedules may slip because of the 
associated learning curve. 


Commercial parts are being selected for space night applications ana weir 
suitability to meet environmental conditions is unknown; these parts may fail 
to operate on-orbit within the environment window, leading to system level 
failures. Abo, environmental testing of these parts can be expensive and cause 
schedule delays. 


i me nign-speea iioer opuc u«ut uus u umniw wvuuuiu^j, ««« 

perform as specified and high data transfer rates will not be sustained. 

First time the IK Instrument Project manager is managing a project to go into 

space; Project may fail due to insufficient / poor management 

Lack of a thorough hardware test program; mission failure due to 
environmental conditions not tested. 

Projecf software schedule and resources were underestimated; Schedule 
slips, cost overruns, and a reduction in adequate testing time are likely results. 


Classifying Risks 


Purpose: 

• look at a set of risks and how those risks 
relate to each other within a given structure 

• efficiently sort through large amounts of data 


Description: 

involves grouping risks based on shared 
characteristics. The groups or classes show 
relationships among the risks. 


NASA SATC 


Rev 2. 1/99 




Classification Perspectives 



NASA SATC 


Rev 2. 1/OT 








Example IR-SIP 
Classification of Risk - 2 


By Impact of Risk 


ll;e''iU;n7^777TTM 


a thorough hardware test program; mission failure due to environmental conditions nottes 


“Mission objectives require - the use of new technology in anms 
approach involves scaling down existing technology to operate at higher frequencies. Manufacturability and 
survivability of the more delicate part is unproven. Problems in either of these areas may result in schedule 

delay, cost overruns, or a shortened mission life. 

"Ability of new hardware to meet sampling rate timing requirements is unknown; failure to meet sample rate 
requirements could result in loss off science data and we may need alternative hardware or be forced to accept 
decreased software performance requirements. 


is is me first time mat tne sonware sum wm use uuu, i ne staff may have a lower-tnan-expectea 
{productivity rate and schedules may slip because of the associated learning curve. 


- F irst time Bie I R Instru ment Project manager is managing a project to go into space; project may fail due to 
insufficient / poor management 

" Waterfa ll l ifecycle model is being us ed to develop all 1K-SIP software; it may cause serious integration 
problems between IR-SIP Cl and IR sensor and/or between IR-SIP Cl and AA platform leading to a missed 
launch window, excessive cost to meet window, or failure to successfully integrate the system. 


Dealing With Sets of Risks 


During classification, it may be decided that 
some risks should be mitigated and tracked as 
a set. When this happens 

• create a summary risk statement 

• assign new ID but maintain linkages to 
original risks 

- keep all context 

- move individual risk statements and ID #s 
to context 

• keep the worst-case impact, probability, and 
timeframe attribute evaluations 

• update database 
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Example - Consolidating Risks 


Probab Im- 


101 

Use of C++, the selected compiler, and OOD are new for software 
staff; decreased productivity due to unexpected learning curves 
may cause coding schedule to slip. 

H 

1 

This is the first time that the software staff will use OOD; The 
staff may have a lower-than-expected productivity rate and 
schedules may slip because of the associated learning curve. 

H 

16 

The C++ compiler selected for use does not come with very gooc 
user documentation, as supplied by the vendor; decreased 
productivity likely as software developers stumble over the same 
problems. 

M 


This is the first time that software staff has used C++; staff may 
have lower-than-expected productivity rate, schedules mav slip. 


Prioritizing Risks 


act frame 


Purpose: 

• sort through a large amount of risks and 
determine which are most important 

* separate out which risks should be dealt with 
first (the vital few risks) when allocating 
resources 


Description: 

• involves partitioning risks or groups of risks 
based on the Pareto “vital few” sense and 
ranking risks or sets of risks based upon a 
criterion or set of criteria 
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Two Step Risk Prioritization 


Master list 
of risks 


Top 10% 


Top 20% 


Prioritized & Ordered 
Master List of 
Top N 
RISKS 


Example Pareto Top 20% 



ability tram* 



Commercial parts are being selected for space flight applications and “ ’ 

their suitability to meet environmental conditions is unknown; these parts 
2 may fail to operate on-orbrt within the environment window, leading to H H M 

system level failures. Also, environmental testing of these parts can be 
expensive and cause schedule delays. 


10% 

Lack of a thorough test program; mission failure due to environmental H FT F 

5 conditions not tested. 



Project resources (personnel number and availability) and schedules were 
100 underestimated; schedule slips, cost overruns, reduction in adequacy of H j H N 

development processes (especially testing time adequacy) likely. 



use of C++, the selectee compiler, and UUU are new Tor software staff; 

101 decreased productivity due to unexpected (earning curves may cause H M N 

coding schedule to slip. 


20% 

Yearly congressional NASA budget profiles~are subject to change; this 
10 may cause the project funding profile to change each year with associated H M F 

replanning, schedule impacts, labor cost increases, loss of key personnel, 
or project termination. 



First time the IK instrument Project manager is managing a project to go i ' 

4 into space; Project may fail due to insufficient / poor management M H N 



science requirements nave substantial i bus; late completion or l bus 
7 likely, with reduction in adequate testing time, possible science applies- M H M 

tion software failure, incorrect science data being captured, hardware 
damage if incorrect safety limits were provided, extensive rework and 
substantial cost overruns, mission failure if problems not found before 
system is in operation. 
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Prioritization Criteria 


The criterion or set of criteria used to rank the 
risks is chosen based on what’s most 
important to the project. 

Recall IR-SIP example: 

• must meet the schedule 

• can’t delete any of the technical or 
performance requirements 

• must keep to the budget 


Exercise 


Multivoting 
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Continuous Risk Management 


Case Study EXERCISE - Multivoting Form 

Directions: It is October 20, 1995. Jerry Johnstone, R.C.Everette, W. Peacock, and C. White have 
come together to prioritize the risks on the Top N list (which were selected using the Pareto Top N 
Method). Review the risk statements and context with respect to the prioritization criteria 

• must meet the schedule 

• can’t delete any of the technical or performance requirements 

• must keep to the budget 

Vote for the three risks that are most important to the project based on the prioritization criteria. Give 
the most important risk 3 points, the next most important risk 2 points, and give the third most 
important risk 1 point. 


Risk 

ID 

Risk Statement 

Points 

2 

Commercial parts are being selected for space flight applications and their suitability to meet 
environmental conditions is unknown; these parts may fail to operate on-orbit within the 
environment window, leading to system level failures. Also, environmental testing of these 
parts can be expensive and cause schedule delays. 


“5 

Lack of a thorough hardware test program; mission failure due to environmental conditions 
not tested. 


“TOD - 

Project resources (personnel number and availability) and schedules were underestimated, 
schedule slips, cost overruns, reduction in adequacy of development processes (especially 
testing time adequacy) likely. 


“TUI 

Use of C++, the selected compiler, and uOl) are new tor software staff; decreased 
productivity due to unexpected learning curves may cause coding schedule to slip. 


1TJ 

Yearly congressionaTNASA budget profiles are subject to change; this may cause the project 
funding profile to change each year with associated replanning, schedule impacts, labor cost 
increases, loss of key personnel, or project termination. 


■ 

First time the IR Instrument Project manager is managing a project to go into space; Project 
may fail due to insufficient / poor management. 


"7 

Science requirements have substantial 1 6 Ds; late completion of I bDs likely, with reduction in 
adequate testing time, possible science application software failure, incorrect science data 
being captured, hardware damage if incorrect safety limits were provided, extensive rework 
and substantial cost overruns, mission failure if problems not found before system is in 
operation. 


14 

Contracting a different test facility for acoustical testing, parts may be insufficiently tested or 
parts may be damaged with excessive testing. 



There is rib AA Satellite Simulator currently scheduled tor development; probable that the IK- 
SIP CSCI will fail when initially integrated with the actual AA Satellite since prior interface 
testing will not have been possible, thus fixes will be done very late in the project schedule 
and may cause the launch date to slip. 



Subset of IR Post Processing CSCI requirements is to be satisfied with COTS products, 
Integration time and lifecycle costs may increase from original estimates which assumed 
significant saving from COTS use, leading to schedule slips and cost overruns. 
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Comparison Risk Ranking 
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Continuous Risk Management 


Case Study 

Risk Information Sheet After Analysis 


ID 11 

Risk Information Sheet Identified: 

11/ 1/ 95 

Priority 10 

Statement 

• It has recently been decided that the Infrared sensors will be 
developed in-house and how they will communicate and how sensor 
data will be processed will be based on assumptions until the detailed 
design is baselined; the accuracy and completeness of those 
assumptions will determine the magnitude of change in the IR-SIP 
Instrument Controller Cl and Infrared Sensing Unit Cl interface 
requirements - it could be minor or catastrophic. 

Probability M 

Impact H 

Timeframe N 

Origin Class Assigned 

K. Green Requirements to: 

Context The AA program is in the Systems Preliminary Design Phase and the IR-SIP project 
software is in the Software Specification Phase. 

• This is the first time these sensors will be used on a NASA mission. They will still be 
under design and definition during the IR-SIP Controller’s software specification through 
implementation phases. Therefore, assumptions about the interface will have to be made in 
implementing the IR-SIP CSCI and if those assumptions are incorrect, then software 
rewrites will be necessary. We do have access to a reasonable set of assumptions and 
information from a contractor who has developed very similar sensors, but again, we don’t 
really feel 100% confident in those assumptions. 


• Problems were not anticipated in the current success-oriented schedule so there is no slack 
time if the impact of the changes is major. Schedule slips, cost overruns, and reduction in 
adequate testing time are all possible if the assumptions prove false. 


• System testing does not begin until very late in the development, so if problems are 

encountered there is usually no time to make changes in the hardware. Therefore, software 
must provide work-arounds for problems encountered. 

Mitigation Strategy 


Contingency Plan and Trigger 


Status 


Status Date 


Approval 


Closing Date 


Closing Rationale 


/ / 
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Key Points - 1 


Evaluate risks at a level that is sufficient to 
determine the relative importance 

Select attribute definitions (e.g. catastrophic 
impact) that make sense for your project. 

Classify risks to help the project understand 
the risks. 

Group related risks into sets to help build more 
cost-effective mitigation plans. 
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Key Points - 2 

Prioritize to determine which risks should be 
dealt with first when allocating resources. 

Prioritize the risks based on the criteria for what 
is most important to the project. 

Communication is central to: 

• defining project evaluation definitions 

• evaluating risks 

• selecting a project classification scheme 

• classifying risks 

• defining prioritization criteria 

• identifying and prioritizing the top N risks 
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Overview 

Planning activities overview 

Planning activities 

• assigning responsibility 

• determining approach 

• defining scope and actions 

Mitigating a set of related risks 
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What Is Planning? 


Planning is the function of deciding what, if 
anything, should be done with a risk. 

Planning answers the questions 

• Is it my risk? (responsibility) 

• What can I do? (approach) 

• How much and what should I do? (scope and 
actions) 


Planning Activities Overview 


Statement of risk 


Context 

Impact 

Probability 

Timeframe 

Classification 

Rank 


Resources 



Master list 
of risks 


[ Classification 
Class 1 Class 2 


Project goals 
and constraints 


• assign responsibility 

• determine approach 

• define scope and actions 


Statement of risk 


Probability 

T i meframe 

Classification 


Plan Approach 


* Consequences may be added 
to the risk statement if not 
already documented 












Risk Information Sheet 


To be completed: 

•Assigned to 
•Mitigation Strategy 
•Contingency plan and trigger 
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•Or “Do I need to act on this risk?” 
















Project Considerations 


What is currently important to the project, management, 
customer, or user? 

Are there critical milestones the project is currently 
facing? 

What limits and constraints do the project, organization, 
group, or manager have? 

What milestones and limits are fixed? flexible? 

What resources are available for mitigation? 

How does this risk fit into the overall project issues and 
concerns? When is the best time to address or mitigate 
a risk? 


Assigning Responsibility 


Purpose: 

* ensure that no risks are ignored 

* make effective use of expertise and 
knowledge within the project when planning 
for risk mitigation 

* ensure that risks are being managed by those 
with the appropriate abilities, knowledge, and 
authority to commit resources for mitigation 

Description: 

* involves reviewing the risk(s) and determining 
who is best able to deal with the risk(s) 
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Determining Approach 


Purpose: 

• ensure you know enough to make an informed 
decision 

• pick an appropriate approach for effective 
management of the risk(s) 

• establish measurable mitigation goals that 
provide a target for evaluating success and 
direction during the development of action plans 

Description: 

• involves reviewing the risk(s) and determining 
the best approach to take 


Action Plan Approaches 


Action plans 
(Approaches/types) 


Research Accept 


Research 

Plan 


Acceptance 

Rationale 


Formal Documented 
Plan 


□ Generic term for the results (action 
plan type) of an approach to 
planning that does not require a 
format documented plan 


Watch 


Tracking 

Requirements 


Mitigate 

1 

Mitigation Plan 
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Action #1 - Research 


Investigate the risk until you know enough to be 
able to decide 

• if it is still your responsibility 

• what to do about it (accept, watch, or mitigate) 

Risk action plan type 

• research plan 
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Action #2 - Accept 

Do nothing. The risk will be handled as a 
problem if it occurs. No further resources are 
expended managing the risk. 

Risk action plan type 
• acceptance rationale 
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Action #3 - Watch 


Monitor the risks and their attributes for early 
warning of critical changes in impact, 
probability, timeframe, or other aspects. 

Risk action plan type 
• tracking requirements* 


"Tracking requirements include indicators for monitoring the 
risk, triggers, or thresholds for taking action, and reporting 
requirements (e.g., how often, by whom, extent of the report, 
and when). 
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Action #4 - Mitigate 

Eliminate or reduce the risk by 

• reducing the impact 

• reducing the probability 

• shifting the timeframe 

Risk action plan type 

• mitigation plan (action item list or task plan) 

• tracking requirements 
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Discussion - Determining Approach 


Risk Statement 


Assigned 

To: 


Science requirements Hava substantial TBDs; Uta complation of TBDs 
likaty, with reduction in adaquate tasting tima, possibla scianca application 
softwara failure, in correct scianca date baing captured, hardware damage if 
incorrect safety limits ware provided, axtansiva rework and substantial cost 
ovamins, mission failure if problems not found before system is in 

operation. 

" Waterfall lifacycle modal is baing used to develop aU 1R-SIP software; it may 
causa serious integration problems batwaan IR-SIP Cl and IR sensor and/or 
between IR-SIP Cl and AA platform leading to a missed launch window, Everette 
excessive cost to meet window, or failure to successfully Integrate the 


The funding and development schedule for the AA satellite Is subject to 
change; IR-SIP schedule slips, costovenuns, and a reduction in adequate 
testing time are likely as unscheduled changes will have to be made to the 
software to match AA project changes. 


Subset of IR Postprocessing CSC! requirements is to be satisfied with 
COTS products; Integration time and We cycle coats may increase from 
original estimates which assumed significant saving from COTS use, 
leading to schedule slips and cost overruns. 



Contingency Plans 


Not all mitigation plans can or should be carried out 
immediately, for example: 

• there may not be sufficient funding at this time 

• other circumstances (such as having the right 
personnel) may not be right 

• it may be a low probability, catastrophic impact risk 
with an expensive mitigation plan 

May be used as Plan B if Plan A fails 

Contingency plans are held in reserve until specific 
conditions are true or certain events occur 

• watch for the conditions and events! 
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Defining Scope and Actions 


Purpose: 

• take a balanced approach in developing 
effective actions to mitigate risks 

Description: 

• involves reviewing the risk(s) and determining 
the appropriate level of mitigation to take and 
the goal of the mitigation 
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Which Type of Mitigation Action? 

What criteria are used to determine when to use 
action item lists and task plans? 

* relative importance of the risk(s) 

* complexity of the issues 

* breadth of expertise required to develop 
mitigation strategies 

* probability and impact of the risk 
(particularly catastrophic) 

* available planning resources (particularly 
personnel) 
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Action Item List vs. Task Plan 


Task Plan 



Risk statement(s) 


Mitigation goal/ 
success measures 


Related Risks 


Due date for task plan completion 


Chosen strategy(ies) 


Specific actions 


Budaet 


Schedule (e.g., Gantt or PERT charts) 


Risk tracking indicators, thresholds, 
reporting frequency 


Contingency strategy, actions, 
and trigger 


Task Plan Components 

Risks 

Staff roles & responsibilities 

Related risks 

Risk tracking requirements 

Specific actions to take 

Due dates & schedules 

Strategy(ies) 

Success criteria 

Cost of strategy/actions 

Mitigation goals 

Contingency strategy and triggers 
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Example 


Task Plan 
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Continuous Risk Management 


Case Study 
Task Plan 


Responsible Person: J. Johnstone (for approval); R.C. Everette (for 

recommendations and implementation) 

Last Updated: 6/7/96 

Origination date: 3/4/96 

Risk Statement 

Risk # 7 

Science requirements have substantial TBDs; late completion of TBDs likely, 
with reduction in adequate testing time, possible science application software 
failure, incorrect science data being captured, hardware damage if incorrect 
safety limits were provided, extensive rework and substantial cost overruns, 
mission failure if problems not found before system is in operation. 

Classification: Requirements 

Related risks: None 

Identified causes 

• inadequate scheduling to allow for requirements definition 

• inadequate civil service and contractor personnel resource planning 

• all of the science requirements are still not available 

Mitigation goals/success measures/criteria 

The goal of this task plan is to 

Complete the science requirements and submit the change for implementation 

WITHOUT slipping the overall development completion date. It is preferable to not 

use overtime or additional resources. 

Chosen Strategies 

The selected strategies to address the key causes and to reach the mitigation goal are 

• to analyze, research, and complete the TBD science requirements, and to submit 
change requests 

• to reprioritize the baselined requirements and reorder the builds to minimize 
impact of TBDs 
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Specific actions 

The following work breakdown structure (WBS) describes the actions that will be 
performed as part of the mitigation plan and identifies who is responsible for completing 
them. This information will also be reflected in a Gantt chart. 

1 .0 Reprioritize the baselined requirements and reorganize the builds to implement 
the high priority requirements first. The likelihood of their changing will be 
factored into the prioritization process. (J. Johnstone) 

1.1 Identify requirements with high probability of change. (R. C. Everette) 

1.2 Identify critical path dependencies among requirements and software 
modules. (R. C. Everette) 

1.3 Build a prioritized list of requirements. (R. C. Everette) 

1 .4 Reorganize the contents and schedule of builds to meet the new priorities. 
(R. C. Everette) 

1 .5 Distribute the changes in build content and schedule to all personnel, and 
tell the customers that no changes to a specific build will be accepted 
once implementation of that build has begun (except for corrections to 
requirements errors that would cause mission failure). (J. Johnstone) 

2.0 Estimate the impact to the schedule for builds and requirements based on the 
projected completion of the TBD requirements. Verify (as much as possible) that 
the new schedule accounts for the anticipated changes. (R. C. Everette) 

3.0 Complete the requirements document for TBD Requirements 38-42 and submit a 
change request. (John Smith/NASA) 

3.1 Estimate the intermediate completion milestones. 

3.2 Report progress weekly. 

3.3 Complete peer review requirements. 

3.4 Submit change requests upon the completion of the requirements. 

4.0 Complete the requirements document for TBD Requirement 73 and submit a 
change request. (John Smith/NASA) 

4.1 Estimate the intermediate completion milestones. 

4.2 Report progress weekly. 

4.3 Complete peer review requirements. 

4.4 Submit a change request upon the completion of the requirements. 
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5.0 Complete the requirements document for TBD Requirement 1 04 and submit a 
change request. (Mary Blue/NASA) 

5.1 Estimate the intermediate completion milestones. 

5.2 Report progress weekly. 

5.3 Complete peer review requirements. 

5.4 Submit a change request upon the completion of the requirements. 

6.0 Complete the requirements document for TBD Requirements 143-149 and 
submit a change request. (Joe Kelley /University Intern) 

6.1 Estimate the intermediate completion milestones. 

6.2 Report progress weekly. 

6.3 Complete peer review requirements. 

6.4 Submit change requests upon the completion of the requirements. 

7.0 Set up a tracking mechanism for change requests and help R. C. Everette 
determine the magnitude of the problem created by change requests. Weekly 
reports will be provided to R. C. Everette. The reports will include the impact to 
the schedule and the resources required to implement each submitted change. 
(J. Johnstone) 

7.1 Design a weekly status report. 

7.2 Set up automated metrics collection and reporting. 
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Risk tracking indicators 

TBD requirements completion: 

Indicator: actual completion dates compared to planned completion dates 

Trigger: a projected 10% schedule slip in the completion of any requirements 

document is cause for review 

Trigger: a projected 25% schedule slip in the completion of any requirements 

document will trigger contingency plan A 

Change request magnitude 

Indicator: the cumulative schedule impact due to the changes (based on submitted 
change requests) 

Indicator: the cumulative resource requirements required to implement the changes 
(based on submitted change requests) 

Trigger: If either the cumulative schedule impact indicator or the cumulative resource 

requirements indicator exceeds their projections by 20%, it will trigger 
contingency plan B 

Budget 

• Planning/oversight: 

J. Johnstone/R. C. Everette: 5 days 

• Completing TBD requirements: 

3 civil servants: 14 weeks 

1 university intern: 7 weeks, $10,000 

• Reprioritizing: 

R. C. Everette: 7 days 

2 team members: 1 day each to review 

• Tracking costs: 

1 civil servant: 3 days to set up automated system; 

R. C. Everette & 

J. Smith: 2 days each to determine tracking measures, 

triggers, report format, and intermediate triggers. 
(Cost to produce weekly reports is negligible) 

• Totals: 

Civil service time: 1 8-person weeks 

University Intern cost: $10,000 

Expected return: The number of errors is projected to decrease by approximately 75%. 
The amount of resources assigned to late requirements changes should decrease 
accordingly by 75%. For this project, the total estimated savings is 50% of the total 
planned budget. The probability for mission failure due to this risk will be eliminated. 
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Schedule (Gantt chart) 


Action 

Start Date 

End Date 

1 

February 15, 1996 

March 15, 1996 

2 

February 15, 1996 

March 15, 1996 

3 

March 1, 1996 

May 7, 1996 

4 

March 1, 1996 

March 15, 1996 

5 

May 24, 1996 

July 15, 1996 

6 

June 1, 1996 

July 21, 1996 

7 

February 15, 1996 

July 21, 1996 



Time ► 
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Contingency strategies, actions, and triggers 


Contingency Plan A: 

Trigger: A projected 25% schedule slip in the completion of any 

requirements document 

Strategy/actions: Authorize contractor overtime to assist civil service (a maximum of 

10 person weeks in contractor time is allowed). Approval by J. 
Johnstone is required. 

Contingency Plan B: 

Trigger: When either the cumulative schedule impact indicator or the 

cumulative resource requirements indicator exceeds its projections 
by 20% 

Strategy/actions: Drop the lower-level science requirements to compensate for the 

estimated development time required to complete the higher- 
priority requirements. 
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Mitigation Goals and Success 
Measures 


Set a realistic , measurable (or verifiable) goal for 
mitigating the risk, for example 

• avoid any changes to scheduled milestones 

• eliminate change requests unsupported by 
funding to implement the change 

Define success criteria — you need to know when 
you’ve succeeded or failed 

For example 

• all current change requests implemented by 
3/1/96 with no change to scheduled 
milestones 

NASA SATC 5_23 Rev 2. 1/99 


Discussion - 

Goals & Success Measures 

Risk 7 - Science requirements have substantial TBDs; late 
completion of TBDs likely, with reduction in adequate 
testing time, possible science application software 
failure, incorrect science data being captured, hardware 
damage if incorrect safety limits were provided, 
extensive rework and substantial cost overruns, mission 
failure if problems not found before system is in 
operation. 

What Goals & Success measures would you look for? 
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Discussion - 

Goals & Success Measures 


Risk 14 - Contracting a different test facility for acoustical 
testing; parts may be insufficiently tested or parts may be 
damaged with excessive testing. 


What Goals & Success measures would you look for here? 
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Continuous Risk Management 

Case Study 

Planning Worksheet 

Planning Worksheet 

Risk ID 7 Responsibility: J. Johnstone 

Risk statement 

Science requirements have substantial TBDs; late completion of TBDs likely, with reduction in 
adequate testing time, possible science application software failure, incorrect science data being 
captured, hardware damage if incorrect safety limits were provided, extensive rework and substantial 
cost overruns, mission failure if problems not found before system is in operation. 

Mitigation goals and constraints (in observable terms) 

Science requirements must be completed and all related change requests submitted for 
implementation. No slipping of the overall development completion date is allowed. Preferable to not 
use overtime or additional resources but if necessary to keep completion date, do so. 

Additional data (e.g., root causes, impacted elements) 


Related risks 

Alternative strategies/actions Estimated costs 


Related mitigation plans 
Strategy evaluation criteria 

Chosen strategy/actions Success measures 


Contingency strategy 


Contingency trigger 
















Planning for Risk Sets 


Mitigation goals can be hierarchical. 

The planning focus should be on high-priority 
or Top N risks in the set. 

Monitoring a set of risks usually requires a set 
of indicators. 
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Mitigating a Set of Related Risks- 1 

Purpose: 

• increase the cost effectiveness of mitigation 
plans by eliminating duplicate efforts 

• avoid conflicting mitigation goals and actions 

• integrate planning efforts and avoid unnecessary 
time developing plans 


NASA SATC 


Rev 2. 1/93 







The result of planning is a documented decision 
about what should be done with each risk. 


The decision is documented in a risk action plan.* 
The types of risk action plans are: 

• research plan 

• acceptance rationale 

• tracking requirements 

• mitigation plan 

- action item list 

- task plan 

* The term “plan” refers to the approach for mitigating a risk and 
does not necessarily mean a formal documented plan. 
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Case Study 

Approaches to Planning and Their Action Plans 


■o 

to 






Completed Planning Worksheet 
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Continuous Risk Management 

Case Study 
Planning Worksheet 

Planning Worksheet 

Risk ID 7 
Risk statement 

Science requirements have substantial TBDs; late completion of TBDs likely, with reduction in 
adequate testing time, possible science application software failure, incorrect science data being 
captured, hardware damage if incorrect safety limits were provided, extensive rework and substantial 

cost ov erruns, mission failure if problems not found before system is in operation. 

Mitigation goals and constraints (in observable terms) 

Science requirements must be completed and all related change requests submitted for 
implementation. No slipping of the overall development completion date is allowed. Preferable to not 

use overti me or additional resources but if necessary to keep completion date, do so. 

Additional data (e.g., root causes, impacted elements) 

Root causes - incomplete definition of reqts in early phases and inadequate scheduling to allow 
completion; poorly planned use of personnel (civil service and contractor); insufficient funding for 
contractor personnel and not enough civil servants to make up for it; science requirements not 
available in early phases. 

Related risks 

none 

Estimated costs 

$70,000 

$10,000 

$105,000 

worst case: $3 -8 million 
1 person week (civil service) 

Related mitigation plans 

none 

Strategy evaluation criteria 

Minimal contractor cost, no completion date slippage 


Alternative strategies/actions 

Initiate an extra contractor task to analyze, complete, research, 
and complete the TBD requirements 

Analyze, research, and complete TBD science requirements and 
submit change requests ASAP - use civil service and contractor 

Authorize contractor overtime until all requirements are 
complete 

Wait and see how bad it gets - slip schedule then if need to (AA 
satellite completion is probably going to be late as well) 

Reprioritize baselined requirements and reorder builds to 
mini mize impact of TBDs 


Responsibility J. Johnstone 
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Chosen strategy/actions - Success measures 

Analyze, research, and complete TBD science All TT^res^^ 

requirements and submit change requesfcsASAP - use withno overtimerequired 

civilservice aid contractor 

Reprioritize baselined requirements and reorder builds to Build order is not impacted by change 
minimize impact of TBDs requests from TBD requirements 


Track progress and use contingency if necessary Management is not surprised by failure o: 

mitigation plan 

Contingency strategy Contingency trigger 

Authorize contractor overtime to assist civil service. Up Weekly status reveals that TBD 

to 1 0 person weeks in contractor time allowed. Approval requirements are not going to be 
by Johnstone required. documented and closed by the due (hues 

Drop lower level science requirements to make iq> for Insufficient time in schedule to complete 
estimated development time required to complete higher all requirements (as calculated by 
priority requirements. projected impact of schedule and resourc 

hits from change requests and current 
progress on implementation) 



Planning Key Points - 2 


Communication 

• Use teams to develop effective plans. 

• Submit plans for approval and review. 

• Determine tracking requirements for risks and 
mitigation plans 

Remember project, manager, user, and customer 
considerations when planning: 

• what is currently considered important 

• fixed or critical milestones 

• project limits and constraints 

• available resources for mitigation 
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Module 6 






Overview 


Overview of tracking activities 

Tracking activities 

• acquire 

• compile and evaluate 

• report status (plan, risk) 
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Risk Information Sheet 


Completed or Updated: 

-Priority 

-Probability 

-Impact 

-Timeframe 

-Status 

-Status Date 


Tracking Risks and Plans 

1. Tracking the mitigation plan will indicate 

* whether the plan is being executed correctly 

* if the plan is on schedule 

2. Tracking the risk attributes will indicate 

* mitigation plan effectiveness 
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Risk Metrics 


Metrics are used to: 

• measure attributes of a risk 

- impact, probability, and timeframe 

- other risk-specific attributes 

• provide meaningful information to enable 
more informed control decisions 

• assess the impact or success of a 
mitigation plan 

• identify new risks 


Acquire 


Purpose: 

* to collect tracking data for a given risk 


Description 

• a process that includes all of the steps 
associated with collecting information about and 
updating the values of risk measures and status 
indicators for watched and mitigated risks 
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Considerations When Acquiring Data 


Status information is only as good as its 
accuracy and timeliness. 

Stale data are more dangerous to decision 
makers than no data at all. 

When a group of indicators is required, all of 
the data must be acquired from the same time 
period. 

Collect the data needed to track the project’s 
risks. Collect only what you need and use what 
you collect. 


Metrics by Life Cycle Phases 


A***® I 

yta^ . — I Detailed 

I P reliminary 

pSardwarel Desi 9 n 


System s 

Requirements ' 
Analysis Des ' 9n 


Software I 

lequirementJ 

reliminary mmmmm—m 
Desi 9 n Detailed 


^ j Code & 
Debug 
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Data Acquisition - Metrics 
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Trigger/Threshold 


A value of an indicator that specifies the level at 
which an action, such as implementing a contingency 
plan, may need to be taken. 


Generally used to: 

• provide early warning of an impending critical 
event 

• indicate the need to implement a contingency plan 
to preempt a problem 

• request immediate attention for a risk * 

Effective if: (\ ^ 

• does not trip unnecessarily y ) 

• is easy to calculate and report 


Percent within Budget 

Over 

| bu dget 

p Within Budget 

ij Feb Mar Apr May ^ Jun ' Jui Aug ' Sap 


Risk 100 : Project resources (personnel number and availability) and schedules 
were underestimated; schedule slips, cost overruns, reduction in adequacy of 
development processes (especially testing time adequacy) likely. 


Under 

budget 


>AT< 
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Data Compilation & Trigger 
Example 


Risk #14: Contracting a different test facility for 
acoustical testing; parts may be insufficiently 
tested or parts may be damaged with excessive 
testing. 

Data to be collected: Vibration testing spectrum 


Trigger: Upper and lower bounds dependent on 
component ==> excessive or insufficient 
testing 

==> potential new risks 


Data Compilation & Trigger 


Sample Vibration Control Spectrum 



t — ' 


■man 

kzbmsss! 

x. 

FT 


/ 

\ 


J 

Insufficient 
testing - possible 
problems not found 


Excessive 
testing - possible 
Parts damage 


Frequency (Hz) 
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Requirements Metrics Example 


Risk # 7: Science requirements have substantial 
TBDs; late completion, inadequate test time. 


Data to be collected: terminology of document 
=> weak phrases, incomplete terms, optional 
terms, TBDs, TBSs, TBAs 

Trigger: >0 on any 


Requirements Metrics Example 
-Text Analysis 



|ggmg3fia 

ItsaufiHsu 


Tool available from: 

6-i8 http://satc.gsfc.nasa.gov 












Testing Metrics Example 


Risk # 100: Project resources and schedules 
were underestimated; schedule slips, cost 
overruns, testing time inadequate. 

Data to be collected: Number open, closed, 
total number (Linear trend to closure) 

Trigger: 

Total number is not as expected on curve 
Closure rate trend will not hit 0 prior to 
milestone 


Testing Metrics Example - 
Trackinq Errors/Faults/Changes 


Cumulative Problem Reports 
Submitted & Closed 


-Cum. Submitted - 3540 
Cum. Closed - 2043 



SSSSSSSSSSSSSSSSSSSSSaSS 
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Process Metrics Example 
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Data Collection Exercise 


Risk 

| Data to be Collected 

#1 This is the first time that the software staff will 
use 00 D; The staff may have a iower-than-expected 
productivity rate and schedules may slip because of 
the associated learning curve. 



# 20 Subset of IR Post Processing CSCI 
requirements is to be satisfied with COTS products; 
Integration time and lifecycle costs may increase 
from original estimates which assumed significant 
saving from COTS use, leading to schedule slips and 
cost overruns. 



#1 2 Resource availability estimates were overly 
optimistic- schedule shows all resources are 
available at the start of each WBS element; schedule 
slips, cost overruns, and reduction in adequate 
testing time are likely. 

j 
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Report 

Purpose: 

* communicate risk status reports to support 
effective decision making 

Description: 

• a process in which status information about 
risks and mitigation plans is communicated 
to decision makers and team members 
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Spreadsheet Risk Tracking 


Documents data in a spreadsheet format, which 
is periodically reviewed 

Provides a concise set of risk and status 
information in a format that is easy to read and 
comprehend 

Supports routine project meetings where risks 
are being reviewed and discussed 
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Example 


Spreadsheet Risk Tracking 
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Case Study 

Spreadsheet Risk Tracking 
EXAMPLE 

IR-SIP Monthly Project Review: Risk Status Spreadsheet — April 1, 1997 

Priority I Risk I Risk Statement Status Probability | Impact I Assigned To 

ID Comments 

1 22 AA Satellite Simulator is being New risk - H H Johnstone 

developed; impacts to current resulted from 

project plan and other mitigation closure of Risk 
plans are unknown but could be 18. 
significant - availability of resources 
to make use of simulator is 

questionable 

2 100 Project resources (personnel New risk 22 has H H Johnstone 

number and availability) and made this worse, 

schedules were underestimated; Key personnel 

schedule slips, cost overruns, had designated 

reduction in adequacy of back-ups in case 

development processes (especially availability slips, 

testing time adequacy) likely. but Simulator 

work negates 

that. 

3 23 Metrics are being reported only on a New risk M M Peacock 

quarterly basis; schedules may slip identified by W. 
and recognition of their slip may be Wills 
too late for effective replanning to 

take place. 

4 7 Science requirements have TBD’s are being M H Johnstone 

substantial TBDs; late completion of analyzed and 

TBDs likely , with reduction in researched. 

adequate testing time, possible Expect 

science application software failure, completion of first 

incorrect science data being set next week. 

captured, hardware damage if 

incorrect safety limits were 

provided, extensive rework and 

substantial cost overruns, mission 

failure if problems not found before 

system is in operation. 

5 11 It has recently been decided that So far the L M Johnstone 

the Infrared sensors will be assumptions we 

developed in-house and how they used continue to 
will communicate and how sensor hold as we 
data will be processed will be based complete 
on assumptions until the detailed prototypes. Only 
design is baselined; the accuracy very minor 
and completeness of those requirement 

assumptions will determine the changes have 

magnitude of change in the IR-SIP resulted so far 
Instrument Controller Cl and and the ripple 

Infrared Sensing Unit Cl interface has been 

requirements - it could be minor or negligible. 

catastrophic. | 
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Priority Risk 
ID 


Risk Statement 


Status 

Comments 


Probability Impact I Assignee 


J ... 

7 

13 

Waterfall lifecycle model is being 
used to develop all IR-SIP 
software; it may cause serious 
integration problems between IR- 
SIP Cl and IR sensor and/or 
between IR-SIP Cl and AA platform 
leading to a missed launch window, 
excessive cost to meet window, or 
failure to successfully integrate the 
system. 

Project plan 
revised for 
incremental life 
cycle. 

Recommendation 
to move to Watch 
negated by new 
risk 22. Revisit 
next month. 

L 

L 

Everette 

• 


and other Top N risks.... 







Commercial parts suitability for 
space applications is unknown; 
parts failure may lead to system 
failure and use of space grade parts 
may cause schedule delays since 
space qualified parts procurement 
have a procurement lead time of at 
least 1 8 months. 

Commercial parts 
appear to be 
working and 
same reliability 
as space 
qualified parts 



Peacocl 

CLOSE 

D 

18 

There is no AA Satellite Simulator 
currently scheduled for 
development; probable that the IR- 
SIP CSCI will fail when initially 
integrated with the actual AA 
Satellite since prior interface testing 
will not have been possible, thus 
software fixes will be done very late 
in the project schedule and may 
cause the launch date to slip. 

Goldman 
authorized 
development of 
simulator on an 
accelerated 
schedule. IR- 
SIP’s project plan 
must be revisited 
to enable us to 
make use of the 
simulator. 
Recommendation 
to close risk and 
open a new risk 
21, accepted. 



Goldma 



WATCH LIST 





W 

101 

Use of C++, the selected compiler, 
and OOD are new for software staff; 
decreased productivity due to 
unexpected learning curve may 
cause design and coding schedules 
to slip. 

Training appears 
to be effective, 
only 2 people left 
to be trained. 
Calls to help desk 
reduced by 80%. 
Use of expert 
from ORB project 
has been 
successful. 
Recommend 
moving this risk 
to Watch 

L 

L 

Everett 






















Priority 

Risk 

ID 

Risk Statement 

Status 

Comments 

Probability 



W 

15 

The funding and development 
schedule for the AA satellite is 
subject to change and cancellation; 
IR-SIP schedule slips, cost 
overruns, and a reduction in 
adequate testing time are likely as 
unscheduled changes will have to 
be made to the software to match 
AA project changes. 

No change 

L 

H 

Johnstone 



and all other risks which are 

not on the top N list and have not 
been accepted or closed. 





















Reporting Schedule 


Reports are generally delivered as part of 
routine project management activities: 

• weekly status meetings 

• monthly project meetings 

The frequency of reporting depends on: 

• the reporting requirements for each risk or 
risk set 

• the manner in which the report will be used 
Exception reporting may be necessary. 


Tracking Sets of Risks 

Risk sets can be tracked as an entity 

If a mitigation plan has been developed for a set 

• the mitigation plan status data are reported 
for the set 

* risks in a set can also be tracked separately if 
the individual risks are important 


NAi 
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Track Activities Overview 


Statement of risk 


Context 

Impact 

Probability 

Timeframe 

Classification 

Rank 

Plan Approach 


Resources 


Action plans 


Risk & mitigation 
plan measure 


Statement of risk 


Context 

Impact 

Probability 

Timefr a me 

Classification 

Rank 

Plan Approach 

Status 

Metrics 


Tracking Key Points 


Tracking reports communicate information required 
for effective control decisions. 

Tracking information and reports can include 
quantitative indicator data as well as more subjective 
information (e.g., recommendations). 

Tracking information is not limited to formal reporting 
mechanisms. 

Informal reporting of risk-related information by all 
project personnel can aid decision making. 

Risk tracking should be integrated with standard 
management practices - risk management should be 
tailored for a project. 
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Control 

Control 

* a process in which decisions are made based 
on the data presented in the tracking reports 

Risks can be controlled individually or in sets. 


NASA SATC 
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What Is Effective Control? 


Assessing the effectiveness of mitigation plans 
Monitoring the quality of plan execution 
Assessing significant changes in risks and trends 
Determining appropriate responses 
Executing the plan of attack 
Communicating above information 

NASA SATC Rev 2. 1/M 

Controlling a Set of Risks 

When a set of risks is being evaluated and its 
trigger is reached, a decision should be made 
about whether to look at individual risks. 

If the risk being closed is a part of a set of 
risks, an informed decision should be made 
either to close the set or to close selected risks 
within the set. 
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Risk Information Sheet 


Completed Items: 
-Approval 
-Closing date 
-Closing rationale 



Evaluate Tracking Results 


Purpose: 

• allow decision makers to identify significant 
changes in risks, to assess the effectiveness 
of mitigation plans, and to accurately 
determine the best courses of action 

Description 

• uses tracking data to re-access project risks 
for trends, deviations, and anomalies 
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Metric Trend Analysis 


The risk management plan can document which 
project metrics to track. 

Trend and data analysis of project metrics can 
be used to identify new risks. 

Trends can be observed through the evaluation 
of successive reports 

* persistent lateness in taking action 

* oscillating priority values 

* significant changes in the number of high 
impact risks or risks of a particular type 


Testing Metrics - Example #1 
Tracking Errors/Faults/Changes 


Given concerns about inadequate Test Resources 
and Schedules, e.g., 

#6 - Project software schedule and resources were underestimated; 

Schedule slips, cost overruns, and a reduction in adequate testing time 
are likely results. 

#21 - Poor communication between the AA Project’s system engineering 
team and the IR-SIP instrument team; substantial errors may occur in 
the interface between the IR instrument and the AA satellite and 
spacecraft integration testing may take longer than planned and 
consume more resources for software changes to correct the problems. 

What approach would you take? What would you 
collect? What trends would you expect to see 
evolve? etc. 
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Testing Metrics - sample solution #1 
Tracking Errors/Faults/Changes 


Open/Closed 


Good Ctasurv Rat* 


Low Ciosura - 
Potential Risk? 


Error Counts 


Code Percent 

Module Count ***** 


ES1CHECK 


EPHEX 


EEEQP 

TRPAR 


ACTHR 

CANAC 

CSAPM 

CSSRD 

MAMUS 

PRADS 

UCVMP 


Trending Metrics - Example #2 


Given concerns about Unstable or Incomplete 
requirements, which metrics might be useful in 
controlling this risk area? 

#7 - Science requirements have substantial TBDs; late completion of 
TBDs likely, with reduction in adequate testing time, possible science 
application software failure, incorrect science data being captured, 
hardware damage if incorrect safety limits were provided, extensive 
rework and substantial cost overruns, mission failure if problems not 
found before system is in operation. 


What would you collect? What trends would you 
expect to see evolve? etc. 
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Requirements Metrics -Sample Solution #2 
Completeness & Volatility Analysis 


Total Numbtr of Retirements 




1 

0 New 
■ Modified 



- n_ 


□ Deleted 




Calendar Quartar 


CRR 

Looks Good ! 
(Stabfe) 


CRR 

Excessive Changes! 
NOT Stable 


Combination of BOTH views indicates risk area - requirements are NOT YET stable 
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Trending Metrics - Example #3 


Recall concerns about inadequate Test Resources 
and Schedules. 

#6 - Project software schedule and resources were underestimated; 
Schedule slips, cost overruns, and a reduction in adequate testing time 
are likely results. 


Another approach to the same risk - completion 
rates 
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T rend Analysis -sample solution #3 
Completion Metrics 



Decide 

Purpose: 

• ensure that project risks continue to be 
managed effectively 

Description: 

• uses tracking data to determine how to 
proceed with project risks 

- close 

- continue tracking and executing the 
current plan 

- replan 

- invoke a contingency plan 
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Action #1 - Close a Risk 


A closed risk is one that no longer exists or is no 
longer cost effective to track as a risk. 

This occurs when: 

• Probability falls below a defined threshold 

• Impact lies below a defined threshold 

• the risk has become a problem and is tracked 

Closure: 

• recommended by person responsible for the risk 

The closure is reported, database is updated and the 
originator is advised 


Action #2 - Continue Tracking and 
Executing Current Plan 


No additional action is taken when: 


• analysis of the tracking data indicates that all 
is going as expected 

• project personnel decide to continue tracking 
the risk or mitigation plan as before 
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Action #3 - Replan 


A new or modified plan is required when 

■ the threshold value has been exceeded 

• indicators show that the action plan is not 
working 

• an unexpected adverse trend is discovered 
This equates to Mid-Course Correction 
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Action #4 - 

Invoke a Contingency Plan 

A contingency plan is invoked when: 

• a trigger has been exceeded 

• some other related action needs to be taken 

The risk and its mitigation plan continue to be tracked 
after the contingency plan has been executed. 
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Example - Continual Data 
Tracking and Analysis (Risk #5 



Rate of Finding Errors 
82% of projected 

Expected totaf 


errors found 

number of errors 

75% of projected 

errors 








Risk #5: Lack of a thorough hardware test program; 
mission failure due to environmental 
conditions not tested. 


This graph would indicate whether acceptable reliability 
has been achieved such that testing could be terminated. 


ft R 5 3 8 

W«tk 


Risk is dependent on scheduled end of testing 
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Execute 


Purpose: 

* implement both the decision made about a risk and 
mitigation plan as well as to ensure that all decisions 
are appropriately documented for future reference 
and historical record maintenance 

* ensure approval and resources are allocated 


Description: 

• the process where control decisions are implemented 


>ATC 
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Mitigation Status Report 
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Continuous Risk Management 


Risk ID : 

□ 


Approach: 


Mitigation Status Report 

Requirements 

Science requirements likely to change 
substantially; reduction in testing time 
leading to possible application software 
failure and extensive rework and 
substantial cost overruns are likely. 

□ Watch □ Accept 



Date: 



Mitigate 


Risk Status: 


Impact (1) 

H 

Probability 

M 

Current Risk Exposure (RE) 

H 

Initial Risk Exposure (RE) 



□ Green 



Yellow 



Red 


Root Causes: 


Description 

Mitigation Summary 

Actions 

Inadequate scheduling to 

Reprioritize baselined 

1,2 

allow for requirements 

requirements and estimate 


definition. 

impact to schedule. 


All of the science 
requirements are still not 
available. 

Complete TBD requirements. 

3, 4, 5, 6 


1 

2 

3 

Actions ^ 

5 

6 




a Actual 
Schedule 


m Estimated 
Schedule 


4 1 a 
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Continuous Risk Management 


Case Study 

Risk Information Sheet After Tracking and Control 

ID 1 1 Risk Information Sheet Identified: 

11/1/ 95 

Priority 7 Statement 

It has recently been decided that the Infrared sensors will be 
Probability M developed in-house and how they will communicate and how sensor 

data will be processed will be based on assumptions until the detailed 
Impact M design is baselined; the accuracy and completeness of those 

assumptions will determine the magnitude of change in the IR-SIP 
Instrument Controller Cl and Infrared Sensing Unit Cl interface 

requirements - it could be minor or catastrophic. 

Timeframe N Origin Class Assigned 

K. Green Requirements to: J. Johnstone 

Context The AA program is in the Systems Preliminary Design Phase and the IR-SIP project 
software is in the Software Specification Phase. 

• This is the first time these sensors will be used on a NASA mission. They will still be 
under design and definition during the IR-SIP Controller’s software specification through 
implementation phases. Therefore, assumptions about the interface will have to be made in 
imp lementing the IR-SIP CSCI and if those assumptions are incorrect, then software 
rewrites will be necessary. We do have access to a reasonable set of assumptions and 
information from a contractor who has developed very similar sensors, but again, we don’t 
really feel 100% confident in those assumptions. 

• Problems were not anticipated in the current success-oriented schedule so there is no slack 
time if the impact of the changes is major. Schedule slips, cost overruns, and reduction in 
adequate testing time are all possible if the assumptions prove false. 

• System testing does not begin until very late in the development, so if problems are 

encountered there is usually no time to make changes in the hardware. Therefore, software 
must provide work-arounds for problems encountered. 

Mitigation Strategy 

[Mitigation goal/success measures: Reduce the probability and impact of incorrect interface 
assumptions to a minimum: estimated low probability and low impact. Ideally, completion of 
prototype tests will show that assumptions we got from EasySensor were correct and there is 
no impact at all.] 

1. Build prototypes of the IR-SIP CSCI software primitives needed to control the 
interface with the Infrared Sensing Unit early in the software requirements phase. 

• Start by 1/10/96. Prototype should contain all the functionality defined by that date 
for the configuration of the Infrared Sensing Unit. Complete by 1/30/96. 

2. Have early interface tests with the Infrared Sensor Unit to confirm functionality and 
control issues. Allocate enough time for software work-arounds to be developed if 
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Continuous Risk Managemen 



2/7/96 


2/4/96 


2/1/96 


First prototype complete 


Approval 


problems arise. 

Mitigation Strategy (cont.) 

. Test of the interface between the two subsystems will be completed by 2/3/96. 

. Second prototype to command the transmission of sensor data from the Unit to the 
IR-SIP CSCI will be started by 2/12/96 and completed by 2/20/96. 

• All subsequent interface tests will be performed by 2/28/96. 

3. Feed information from the two prototype tests into updates to the Interface 
Requirements Specification and the associated sections of the schedule by 3/2/96. 

4. Determine the impact of the revised requirements by 3/6/96. 


Status 

Interfac e tests in progress - no significant difficulties found so far. 
Expected completion of tests on 2/26 

Second prototype complete 

Testing of the interface complete, ran a bit late tat no significant 


Contingency Plan and Trigger 

Trigger. If the 2/12/96 or 2/28/96 dates cannot be met, put the contingency plan in place. 


Contingency Plan : Elevate this as one of the top 10 project risks and request that project 
reserves be used to pay for additional contract support to get the two sets of 
requirements firmed up (i.e., configuration and data transfer). If additional contract 
resources are not available, slip the schedule for completion of the prototypes to be 
done by March 20, and request that project reserves be used to pay for additional 
resources to be added to the software design and implementation to make up the 
schedule slip. 
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Control Key Points 


• Control Decisions are based on current 
information as well as experience and are 
required to respond to changing conditions 
in watched and mitigated risks. 

* Risk tracking and control should be 
integrated with standard project management 
practices - risk management should be 
tailored for a project. 


NASA SATC 7 ~ 26 Rev 2, 1/99 





Module 8 

Communicate & 
Document 
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Overview 

What is communication? 

Relationship to other paradigm functions 
Enablers to communication 
Barriers to communication 
Types of Documentation 
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Relationship To Other 
Paradigm Functions 



Makes risks, plans, actions, concerns, exchanges, forecasts, and 
progress known 

Ensures the visibility of risk information 

To enable all project members to participate in defining and 
managing risks 

Ensures understanding of risks and mitigation plans 

Establishes an effective, on-going dialog between the manager 
and the project team 

Ensures appropriate attention is focused on issues and concerns 
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Enablers to Communication 


• Defining clear project roles and responsibilities 

• Making risk actions and decisions visible 

• Being a role model 

• Establishing an internal champion 

• Rewarding positive behavior 


Barriers to Communication 


• Ready-fire-aim 

• Don’t tell me your problem 

• Shoot the messenger 

• Liar’s poker 

• Mistrust 

• Value differences 

• Hidden agendas 


• Differential knowledge is 

L JL\ 

• Placing blame mm 

\ 

• Inactive listening 

Ji_A 

NASA SATC _ __ 5-6 
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Types of Documentation 
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Module 9 


HowTo Implement 
Continuous Risk 
Management 


_2J_ 


Overview - 

How to implement CRM 

Frequently Asked Questions: 

- When do I start? 

- Who’s involved? 

- What do I need? 

- What should I choose? 

- What actions should I take? 


NASA SATC 


Hints and Tips 
Things to Watch Out For 


Track 
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When do I Start CRM? 


Opportunity 


Actions 


Pre-contract activity 


Include risk management provisions 
in the solicitation and statement of work. 


Major project 

milestones 

(e.g., contract award) 


Prepare for a major project decision point 
and the need to increase knowledge about 
risks for improved strategic planning. 


Major project review 


Prepare for standard reviews, such as 
design reviews, functional tests. 


Best time to start is at the beginning. Risk information can help in 
planning and budgeting. 


Who’s Involved? - 1 


/ 


Role/Description 


Responsibilities and Tasks 


Sponsor 

(e.g., senior mgr., VP, 
division chief) 

- Provide visible support and encouragement 
• Reward effective management of risks 

Project manager 
(responsible for ultimate 
success of project) 

• Provide resources and funding 

• Reward effective management of risks 

• Monitor progress 

Champion 

(advocates new technology 
or process within the project) 

• Publicize and promote CRM 

• Coordinate changes and improvements on 
the project 

Change agents 
(plan and implement 
changes in organizations 
and projects 

• Assist with recommendations of plans 

• Evaluate existing and new tools 

NASA SATC 
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Who's Involved? - 2 




Role/Description 

Responsibilities and Tasks ^ 

Facilitators 

(trained in meeting skills, 
conflict resolution, tools, 
etc., - act individually or as 
a team) 

• Conduct training sessions 

• Provide CRM expertise 

• Provide consulting during evaluation of 
progress 

Technical managers (e.g., 
team or functional leads, 
such as software/hardware 
manager, test mgr., etc.) 

• Encourage and support use of CRM within 
their teams 

• Report risk information to project manager 

• Evaluate progress within their teams 

Project personnel 
(e.g., software or hardware 
engineers, testers, etc.) 

NASA SATC 

* Add CRM activities to day-to-day 
operations 

• Maintain open communication about risks 

Rev 2. 1/99 


You need an . . . 

Organization Structure 


Example: 


Quality 

assurance 

manager 


Configuration 

management 

lead 


Software 

manager 


Project 

Manager 


Hardware 

manager 


System 

engineer 

manager 


Integration/ 
test manager 


Software 

engineers 


Engineers 


Testers 


NASA SATC 
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You need . . . 

Internal Communication 


Example: 


Project 

manager 


Technical 

leads 


Control 
■ review 

- reprioritize 

- integrate 
across teams 


Assign 

responsibility 


Analyze 

- review 

- prioritize 
“-'evaTuate" 

- classify 


Plan 

- approve 

...plan.?. 

- recommend 

- accomplish 


Individuals/ Risks 


Status/ 


Identify 

◄ 

forecast 

Track 


Status/trends 


You need . . . 

External Communication 


Senior Managers 

Multi-project * 
Integration 


Customer 

Awareness 

Issue 

resolution 


Project 



Decisions, 


Mitigation plans, 


Agreements 


Status reports 

NASA SATC 
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You need . . . 

Established Meetings 

Weekly Team Meetings 

•establish priority of team’s risks 

• assign responsibility for new risks 

• review and approve mitigation plans 

Monthly Project Meetings 

• Leads present the team’s Top N risks (and mitigation 
plans) 

• Project manager Leads decide on appropriate action 

• Project manager determines allocation of resources 
for mitigation discretionary funds for technical leads 
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You should carefully . . . 

Adapt to Your Project 


Purpose: 

• make maximum use of existing, effective 
project management processes and methods 
while integrating a set of proactive risk 
management activities 

• document the tailored processes, methods, 
and tools in a risk management plan 

• define a schedule for transitioning specific 
methods, tools, and activities into the project 

Description: 

• tailors risk management processes, methods, 
and tools for use on the project 


You must choose your . . . 

Risk Database 


* A database is the simplest means of retaining 
and keeping risk information up to date. 

* Data entry forms and reports can be used as the 
risk information sheet, spreadsheet, and other 
templates. 

* A risk database enables documentation of 
lessons learned, trend analysis, pattern analysis 
to support identifying common risks (and 
solutions) across projects. 
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• Based on experience and the SEI Continuous Risk 
Management Guidebook 

• Contains most of the items found on the SEI “Risk 
Information Sheet” 

• Further information available from: 
http://www-osma.lerc.nasa.gov/srmd/riskO.htm 

• Database can be downloaded from the net if you 
have Access 7.0 or greater 


First - 

Document Your Plan 




Risk Management Plan: 

• Overview of Risk Management process 

• Project Organization & Responsibilities 

• Risk Management activities in detail 

• Budget, resources & milestones for risk 
management activities and risk mitigation 

• Procedure for documenting risks 


NASA SATC 
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Exercise - 

Sample Risk Management Plan 


IR-SIP’s Risk Management Plan can serve as DID, 
with example text, for your project to use. 

Take 10 Minutes to read the IR-SIP risk Management 
Plan, then we will walk through it 


Risk 

Management 

Plan 


Second - 

Document Implementation 


Risk Implementation Plan : 


Rig 


JF 

* Sponsorship I 

* Project Organization & Responsibilities for I 

transition l 

* Risk Management activities in detail 

* Budget, resources & milestones for transition effort 

* Evaluation measures and completion criteria 

* Transition risks and mitigation plans 

* Establish baseline method 


NASA SATC 


Rev 2. 1/99 






Continuous Risk Management 


Case Study 

IR-SIP Risk Management Plan Outline 


Baselined: 

Last Modified: 

Owner: 

Purpose: 

Section 1. Introduction 

1. 1 Purpose and Scope 

1.2 Assumptions, Constraints, and Policies 

1.3 Related Documents and Standards 
Section 2. Overview of Risk Management Practice 

2. 1 Overview 

2.2 Process and Data Flows 

2.3 Project Management Integration (optional) 

Section 3. Organization 

3. 1 Organizational Chart 

3.2 Project Communication and Responsibilities 

3.2 A A Program Responsibilities 

3.3 Contractor Responsibilities 
Section 4. Practice Details 

4. 1 Establishing Baselines and Reestablishing Baselines 

4.2 Identifying Risks 

4.3 Analyzing Risks 

4.4 Planning Risks 

4.5 Tracking and Control of Risks 

4.6 Summary of Methods and Tools 

Section 5. Resources and Schedule of Risk Management Milestones 
Section 6. Documentation of Risk Information 
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Case Study 

IR-SiP Risk Management Plan 


Baselined: 11/15/95 

Last Modified: N/A 

Owner: J. Johnstone/IR-SIP Project Manager 

Purpose: This plan documents the practice of risk management as tailored to the IR- 
SIP Project. This plan will be updated on 2/25/96 and 4/25/96 to reflect changes and 
improvements to the risk management practice based on the evaluation results. 

Section 1. Introduction 

This plan will direct the processes, methods, and tools used to manage risks in the IR- 
SIP Project. All project personnel are responsible for following this plan. This plan is 
part of the IR-SIP Project Management Plan suite of documents. 

1.1 Purpose and Scope 

This plan will define the practice of risk management as it should be performed once it 
reaches maturity within the IR-SIP Project. This document does not address risk 
management within the AA Program. 

1.2 Assumptions , Constraints , and Policies 

This plan does not address the process of putting a new risk management practice in 
place (in other words, the actual transition process - that is documented in the 
Implementation Plan). This plan defines the risk management practice for the IR-SIP 
Project. It is recognized that this plan addresses a new practice being put into place on 
a project that is already in progress and that this plan is the first of its kind for IR-SIP .It 
is expected that significant changes and improvements will be necessary over the 
course of time as risk management is adopted by IR-SIP. Therefore, any corrections 
should be forwarded to the plan owner. Change recommendations should be submitted 
on the Change Documentation Request Form 1246. 

1.3 Related Documents and Standards 

IR-SIP Risk Management Implementation Plan will guide the technology transition 
process. It directs the flow of activities associated with getting the risk management 
practice defined in this plan established and ongoing. 

IR-SIP Project Management Plan directs the activities of the overall project. The Risk 
Management Plan is subordinate to project management plan. 
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Section 2. Overview of Risk Management Practice 
2.1 Overview 

This section provides an overview of the risk management practice and its relation to 
IR-SIP’s project management. Details are to be found in the following sections. The 
overview of the process will be defined by a process/data flow diagram. 

There are four primary activities performed in the risk management practice: 

• identification of risks: a continuous effort to identify and document risks as they are 
found 

• analysis of risks: an estimation of the probability, impact, and timeframe of the risks, 
classification into sets of related risks, and prioritization of risks relative to each 
other 

• planning risks: decision about what to do with the risks, which, for important risks, 
will include mitigation plans 

• tracking and controlling risks: collection and reporting status information about risks 
and their mitigation plans (where appropriate) and taking corrective action as 
needed. 

The risk management activities will be carried out during day-to-day activities of project 
personnel as well as during key project meetings. 

Only Top 20% risks shall have any resources expended for mitigation. All non-Top N 
risks shall be watched or accepted. 
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2.2 Process and Data Flows 

The following diagram depicts the overall process of managing risks on the IR-SIP 
Project. 


Individual/Daily activities 


First Level Manager activities 


(identity risks ) — 


Analyze: 

Classify risks 
Evaluate risl 


l r 



Analyze 

Prioritize 


aze risks 7 n develop actiori | 



\Miich risks to report up - 
Software resource authorization | 
Decisbns: 

- dose risk - 

- replan 

- continue tracking 
• take planned action 



Monthly IR-SIP Project Meetings 


{ Control: 

Project risks 


Project resource authorization 

Reassigned risks 

Top N risks 

Decisions: 

- dose risk 

- replan 

- continue tracking 

- take planned action 


Monthly AA Program Review 


Decisions: 

- reassign risk 

- dose risk 

- authorize program resources 


2.3 Project Management Integration (optional) 

The IR-SIP Project Management Plan calls for the identification, processing, and 
documentation of changes and problems to the system. Risks will, in general, be 
considered an equivalent item to problems and changes in terms of tracking and 
significance during project meetings. Top 20% risks will be handled similar to critical 
issues, as documented in the Project Management Plan. Any risk which is also a safety 
risk will be handled similar to a safety-related problem - referral to the project’s safety 
plan or to the Safety Guidebook NASA-GB-1 740.1 3-96. 


9b-4 












Continuous Risk Management 


Section 3. Organization 
3.1 Project Organization 

The IR-SIP project organization is defined in the Project Management Plan and 
repeated here for convenience. 
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3.2 Project Communication and Responsibilities 

The following diagram introduces the structure of risk communication and responsibility 
within the IR-SIP organization for conducting risk management activities. 



The responsibilities of all project personnel as individuals, the team or technical leads, 
the function leads, and the project manager are specified in the following table. This 
table illustrates the type of responsibilities that need to be identified and allocated to the 
project personnel for the management of risks. 
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Who 

Responsibilities 


Software/Hardware engineers, testers, leads, and project manager 

• identify new risks 

• estimate probability, impact, and timeframe 

• classify risks 

• recommend approach and actions 

• track risks and mitigation plans (acquire, compile, and report) 

S/W, H/W, CSCI, 
CM, and Test 
Managers 

Leads for each esCi 

• ensure accuracy of probability/impact/timeframe estimates and the classification 

• review recommendations on approach and actions 

• build action plans (determine approach, define scope & actions) 

• report their Top N risks and issues to the project manager 

• collect and report general risk management measures/metrics 

Software Project 
Manager, 
Hardware 
Project 
Manager, etc. 

integrates risk information from all technical leads 

• reprioritizes all risks to determine Top 20% risks in each area (software, hardware, 
etc.) 

• makes control decisions (analyze, decide, execute) for risks (e.g., Software Project 
Manager controls software risks) 

• authorizes expenditure of resources for mitigation 

• assigns or changes responsibility for risks and mitigation plans within the CSCI, 
CM, and test areas 

• handles communication IR-SIP project manager 

IR-SIP Project 
Manager, 
IR-SIP Project 
Systems 
Engineer 

"• integrates risk information from all software, hardware, and CM leads 

• reprioritizes all risks to determine Top 20% project risks 

• makes control decisions (analyze, decide, execute) for Top 20% project risks 

• authorizes expenditure of project resources for mitigation 

• assigns or changes responsibility for risks and mitigation plans within the project 
(e.g., moving responsibility for a risk from software to hardware) 

• handles communication with AA program manager 

• reviews general risk management measures/metrics with Quality Assurance during 
each quarter to evaluate effectiveness of risk management 


The criteria for communicating risk information is documented in the following table. 


Communication Path 

Criteria for Selecting Risks and Status Information 

Technical leads to Jerry 
Johnstone 

• Top 20% risks for each team 

• Any risk that impacts launch readiness 

• Any risk with an impact >10% of budget 

• Any risk that needs to be transferred to another team 

Jerry Johnstone to AA Program 
Manager (Goldman) 

• Top 20% risks in the project 

• Any risk that impacts the satellite’s operation 

• Any risk with major impact on IR-SIP operations 

• Any risk that impacts the launch schedule 

• Any risk that exceeds 25% of the project budget 

• Any risk that negatively impacts NASA’s reputation 

Everette to contractor program 
manager 

• Any risk that impacts the contractor’s ability to succeed 

• Any risk that impacts the overall project schedule 

• Any risk that needs to be transferred or jointly managed by the 
contractor 

Jerry Johnstone to Program 
System Engineer 

• Any risk that impacts the satellite’s operation 

• Any risk that impacts the launch schedule 

• Any risk that exceeds 25% of the project budget 

• Information on technical problems that affect the spacecraft or 
other instruments 
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3.3 AA Program Responsibilities 

If IR-SIP project personnel identify risks that affect the AA Program, it is the 
responsibility of the IR-SIP Project Manager to notify the AA Program Manager. The AA 
Program Manager, with the assistance of the change agent P. Stone and the IR-SIP 
Project Manager, to manage risks transferred to the SE Program level. 

The IR-SIP Program manager shall report progress summaries on Top N IR-SIP risks to 
the AA Program Manager on a monthly basis. The AA Program Manager is responsible 
for authorizing additional expenditures if requested by the IR-SIP Project Manager and 
transferring assignments of risks to the IR-SIP Project. 


Project 
Top N 
risks 



Decisions, 
assignment of 
risks 


Meeting 


Monthly ana 
major milestone 
AA Program 
Manager reviews 


Purpose 

IR-§|p, CAM-SIP, SPEC-SIP, AA Spacecraft Project 
Managers, their Systems Engineers, AA Program Systems 
Engineer, and Safety & Environment Mission Assurance 
Manager meet with AA Program Manager to review 
program status and issues. 

Risk-specific information from each project 

• new Top 20% risks, any risks that impact the program 

• status of safety risks 

• status of all Top 20% risks 

Status for program risks are reported by the program 
manager. 

Decisions and actions include 

• decisions/resolutions for risks that are not being 
successfully managed 

• approval for mitigation plans and resources that exceed 

normal project limits 


Method or Tool 


Risk Information 
Sheets 

Cost-Benefit 

Analysis 

Safety 

risk/hazard 

information 


9b-8 






Continuous Risk Management 


3.4 Contractor Responsibilities 

Software Contractor reports to the Software Project Manager. Since the original 
contract did not call for risk management, risk management performed by the contractor 
and reported to the Software Project Manager is voluntary. Contractual modifications to 
install risk management as a part of the contract would result in an update to this part of 
the Risk Management Plan. 

Section 4. Practice Details 

This section provides the details about the practice needed to enable project personnel 
to carry out the risk management activities. 

4.1 Establishing Baselines and Reestablishing Baselines 

A baseline set of risks was established before this plan was written. That baseline shall 
be updated or re-established periodically at major project milestones. Risk baseline re- 
establishment is conducted using the following process. 



' Action 

i 

IR-SIP project manager identifies a cross section of project personnel. All levels and 
disciplines should be represented in this group. 

2 

Group uses the TBQ Interview method to generate risks in a two-hour session. 

3 

Group evaluates risks using the Tri-Level Attribute Evaluation method. 

4 

Group classifies according to source in the Risk Taxonomy. 

5 

Project Managers and Functional Area Managers prioritize to identify the Top N risks or sets of 
risks. 

6 

Project Managers and Functional Area Managers compare Top N risks from this effort to 
existing Top N risks. Expand project Top 20% risks list to include the rebaselining Top N. 

7 

Project Managers and Functional Area Managers reprioritize new Top N. 

8 

Assign new Top N risks to personnel to begin building action plans. 

9 

Add all other rebaseline risks to the database and determine which ones will need to be 
transferred, delegated, watched, accepted, or researched. 

10 

PM distributes rebaseline set of risks listing to the project and asks for additional information 
from anyone in the project who might know more than what is documented. 


4.2 Identifying Risks 

All personnel are responsible for identifying new risks. The database can be accessed 
by anyone at any time to identify new risks. The Short TBQ and project data shall be 
reviewed twice per month by all project personnel to help identify new risks. Project 
metrics (as defined by the Goal/Question/Metric method) will be reviewed whenever 
any predefined thresholds or triggers are reached that would indicate a potential 
problem (i.e., a risk). Risk statements shall be written according to the format, 
“condition; consequence.” All relevant information shall be captured as context. The 
risk database shall automatically assign a risk identifier and tag the identifier’s name 
onto the report. The Risk Information Sheet shall be used as the input form for risk 
information. 

Any new risks identified during any project-related meeting shall be added to the 
database within two working days of the meeting. It is the responsibility of the meeting 
leader to make sure that this is accomplished. 
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[Note to students: The actual procedure steps for accomplishing this task would go here 

- equivalent to the procedure steps listed for re-establishing a baseline.] 

4.3 Analyzing Risks 

Risk attributes of probability, impact, and timeframe shall be estimated by the identifier 
of the risk and entered at the same time the risk is identified. If the identifier does not 
know the value of the estimates, it can be skipped during database entry. The team 
mangers shall be responsible for reviewing and correcting attribute values for new risks 
on a weekly basis. 

The Tri-Level Attribute Evaluation method shall be used for evaluating attributes. 
Classification shall be done using risk source according to the Risk Taxonomy. 
Prioritization shall be accomplished noting that only the Top N risks shall receive 
mitigation resources. Determination of the number of Top 20% risks to maintain shall be 
made by the PM and FAMs for the project and the functional area. 

[Note to students: The actual procedure steps for accomplishing this task would go here 

- equivalent to the procedure steps listed for re-establishing a baseline ] 

4.4 Planning Risks 

All Top 20% risks shall be assigned to someone within the project for responsibility. 
Accomplishment of actions contributing to the mitigation of the risk may be assigned. 
Responsibility for a risk means that the responsible person must answer for the status 
and mitigation of the risk. 

Assign responsibility: As newly identified risks are brought to a manager’s immediate 
attention through weekly database reports, the manager shall determine whether or not 
to keep the risk, delegate responsibility, or transfer responsibility up the project 
organization. If transferred, the transferee must make a similar decision. The project 
manager, if necessary, can transfer a risk to the AA Program Manager. 

When you are assigned or keep responsibility for a risk: Decide if the risk requires 
further research (then create a research plan); accept the risk (document acceptance 
rationale in the database and close the risk), watch (define tracking requirements, 
document in the database, and assign watch action), or mitigate (create a mitigation 
plan, assign actions, and monitor the plan and the risk). See Appendix A for standard 
plan templates. Note that only Top N risks shall be mitigated. 

Mitigation plans shall be either an action item list or follow the standard template for IR- 
SIP task plans. Task plans shall be written for any mitigation effort that requires 
reallocation of project resources. The project manager shall determine when to use a 
task plan format. 

[Note to students: The actual procedure steps for accomplishing this task would go here 

- equivalent to the procedure steps listed for re-establishing a baseline.] 

4.5 Tracking and Control of Risks 

The person responsible for a risk shall provide routine status reports to the Functional 
Area Managers and PM during weekly Functional Area meetings and the weekly and 
monthly project meetings. The status for each Top 20% risk shall be reported each 
week in their respective meetings. Status on all watch lists shall also be reported during 
the monthly meetings. The Risk Spreadsheet shall be used to report summary status 
information for risks. The Stoplight Status Report shall be used by the PM to report 
progress to the AA Program Manager at the program monthly reviews. 

[Note to students: The actual procedure steps for accomplishing this task would go here 

- equivalent to the procedure steps listed for re-establishing a baseline] 
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4.6 Summary of Methods and Tools 



TJsei I 

Risk Information Sheet 


Problem-Solving Planning 

Used for developing mitigation plans tor complex risks. 

Penodic review ot project data and the Short 
TBQ 

Used tor routine or frequent identification of risks. I he 
short TBQ provides a memory jogger for possible 
sources of risks and the project data is reviewed with 
that list in mind. 

Goal/Uuestion/Metric tor project metrics 

Use project metrics to help identity and track risks. 

Action Item Lists 

Used tor developing a list of relatively simple mitigation 
actions. 


Used technical leads to succinctly report current status 
information about their teams’ risks. 

Taxonomy Classification 

Used when risks are identified as a structure tor 
grouping related risks. Technical leads use this to help 
eliminate duplicate risks and combine related mitigation 
plans. 

■■ 

Used when risks are identified to evaluate probability, 
impact, and timeframe. Also helps level the risks into 
those that might be important enough to be considered 
Top 20% risks (filters out the less important risks). 
Safety risks are evaluated according to the Safety 
Handbook. 

Multivoting 



Section 5. Resources and Schedule of Risk Management Milestones 

Resources for the management of risks are broken into two categories: 

• overhead costs associated with the risk management process: 00.05% of the project 
budget 

• mitigation plan costs: resources associated with mitigation plans, specifically those 
with task plans 

Budget allocation for mitigation plan development and execution is initially set at 1% of 
the project budget, with equal portions of that distributed to each functional area. Each 
Functional Area Manager is responsible for managing their mitigation budget. Any 
requirements for additional mitigation resources must be made to the Project Manager. 

Milestones 

• Weekly project and functional area meetings shall include statusing of risks. 

• Monthly project meetings shall include statusing of risks. 

• Top 20% risk status shall be summarized and reported to the AA Program Manager 
on a monthly basis. 

• The baseline set of risks shall be re-established on a project milestone basis. 

Section 6. Documentation of Risk Information 

All risk information shall be documented in the risk database. The risk database is 
accessible by all project personnel for the purpose of identifying new risks. Once a risk 
has been assigned to someone, then only that person shall have the authority to 
update the risk information. The Risk Information Sheet for any risk can be printed by 
whomever is assigned to the risk. Spreadsheets and Stoplight Status Reports can only 
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be printed by the Program Manager, Functional Area Managers or their designated 
assistants. 

The responsible person must document lessons learned before closing the risk. Those 
lessons learned must be reviewed and approved by whoever is assigned closing 
authority for the risk before the risk can be officially closed within the database. 

The IR-SIP database is being provided at no cost by the SR & QA office. Assistance in 
maintaining and modifying the database is also being provided at no cost, provided it 
does not exceed two hours per week. Any additional needs must be negotiated 
between the IR-SIP PM and the SR & QA director. 
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Exercise - 

Sample Implementation Plan 


IR-SIP’s Implementation Plan can serve as DID, with 
example text, for your project to use. 

Take 5 Minutes to read the IR-SIP Implementation 
Plan, then we will walk through it. 


Implementation 

Plan 


http://arioch.gsfc.nasa.gov/302/Risk/RMPage.htm 
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Case Study 

Implementation Plan 

for Installing 

Risk Management Practice in IR-SIP 

Baseline Date: 

Last Modified: 

Owner: 

Co-owners: 

Purpose: 

Section 1 . Sponsorship 

1.1 Sponsorship Roles and Responsibilities 

1.2 Reporting Requirements 

1.3 Sponsorship changes 
Section 2. Roles and Responsibilities 

2. 1 1nfrastructure Roles to be Filled 
2.2 Project Personnel roles 
Section 3. Schedule of Activities 

3. 1 Detailed Transition Schedule Milestones 

3.1.1 Basic Risk Management Practice Phase 

3.1.2 Improvement Phase 

Section 4. Allocated Budget and other Required Resources 

Section 5. Evaluation Measures and Completion Criteria 

Section 6. Risks and Mitigation Strategies for this Implementation Effort 

Section 7. Establish Risk Baseline Method 
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Case Study 

Implementation Plan 

for Installing 

Risk Management Practice in IR-SIP 1 


Baseline Date: 9/10/95 

Last Modified: 2/1/96 


Owner: 


J. Johnstone/IR-SIP Project Manager 


Co-owners: R. Douglas/Manager SR & QA Office 

Purpose: This plan documents how the practice of risk management will be designed 
and installed into the IR-SIP project. It does not specify what IR-SIP’s actual risk 
management practice is, only the process for putting it in place. 


Section 1. Sponsorship 

Sponsorship for this effort is being supplied by Jerry Johnstone, as project manager for 
IR-SIP; Stu Goldman, as program manager for the AA Program; and R. Douglas, 
manager of the SR & QA office of this organization. Expansion of risk management 
into the rest of the AA Program is dependent upon the success of the IR-SIP 
implementation. 

1.1 Sponsorship Roles and Responsibilities 

The sponsors shall provide continual, visible support for this effort at all levels of the 
organization. This shall include the following: 

• Goldman’s report of status at the quarterly site Management Review 

• All three sponsors’ written endorsement and encouragement of this effort to all IR- 
SIP project personnel 

• All three sponsors’ attendance at first kick-off meeting with IR-SIP personnel and 
periodic attendance at IR-SIP Monthly meetings 

• Monthly status meetings held with all three sponsors and change agent P. Stone. 

• All sponsors shall allocate budget to this effort as specified in Section 4. 

• Any further supportive announcements or activities as recommended by P. Stone. 

1.2 Reporting Requirements 

The IR-SIP Project Manager shall make monthly progress reports on the 
success/difficulties of implementing risk management (see Section 6, Risks and 
Mitigation Strategies for this Implementation Effort). Requests for assistance from SR & 
QA in the form of training, process definition and improvement, etc. should be made on 
an as-needed basis. Status reports shall include evaluation of progress measures of 
the implementation effort as well as a summary listing of all risks in the project. Use of 
the center-standard risk database is required. Roll-up of all project risk data into the 
center database is required on a quarterly basis. 


1 Note: Another name for an implementation plan is transition plan. 
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1.3 Sponsorship changes 

In the event of personnel changes in the sponsors, this implementation plan must be re- 
evaluated and reapproved. Summary reports of progress to date may be required from 
the project manager. 


Section 2. Roles and Responsibilities [updated 9/20/95] 

This section identifies the roles and associated responsibilities for this transition effort. 

Note that one person may fulfill multiple roles. Sponsors were identified in the previous 

section. 

2.1 Infrastructure Roles to be Filled 

These roles need to be filled in order to support the transition of risk management into 

the IR-SIP Project. The same personnel may be required to continue these roles if risk 

management is later rolled out to other parts of the AA Program. 

• Champion: Someone from within the project, preferably from the managerial level, to 
provide motivation and leadership. This person will be responsible for encouraging 
and reinforcing the proper management of risks and open communication of risks as 
part of his/her routine activities, and assisting in the periodic evaluation of this 
transition effort. 

• Assigned to: R.C. Everette 

• Change agent: Expected to be provided from the local software working group 
representatives (must be from outside the project/program). This person will be 
responsible for coaching project personnel in the accomplishment of risk 
management activities. Estimated time requirements are 10 hours per week, on 
average. Will also train project personnel in the tailored risk management practice 
and assist the champion and program manager in locating tool training (as needed). 
This person should have training and leadership skills. 

• Assigned to: P. Stone (SWG member) 

• Facilitation team: Require two from outside the project and two from inside the 
project. Facilitation skills are needed or training must be provided. At least two 
members of this team should be experienced facilitators. Facilitators will be called 
upon to help the project whenever facilitation is required to handle issues or carry 
out specific methods or procedures that require a facilitator. This team will also 
assist in the establishment of the risk baseline. Estimated time commitment: 
Baseline establishment - two person weeks each; routine assistance - one 
hour/week each (on average) but expect a higher peak in early phases. 

• Assigned to: P. Stone, J. Douglas/SWG; Blue/software engineer, L. Jason 
(quality assurance). All four of these individuals are already trained facilitators 
and have committed their time and effort. Everette and M. Jones have agreed 
to allow the IR-SIP project individuals to fulfill these roles. Stone’s and 
Douglas’ managers have also committed to supporting this effort. 


2.2 Project Personnel roles 

These are the roles and responsibilities of the IR-SIP project personnel. 

• Take risk management training: When training in tailored risk management practice 
for IR-SIP is made available, all project personnel are expected to take the training. 
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Schedule allowances will be made by the project manager to accommodate near- 
term deadlines. 

• Conduct risk management activities: Project personnel are expected to carry out the 
risk management activities that are defined in the IR-SIP Risk Management Plan 
once it has been generated. 

• Facilitation team members (see Section 2.1): Two project personnel will be assigned 
to this team. Work allocations will be adjusted by management to accommodate 
duties. 

• The initial entry of baseline risk information into the database shall be performed by 
G. Whitley under guidance of a SR & QA representative and the change agent, P. 
Stone. 

• It is expected that all project personnel will participate in the performance of risk 
management activities. Data entry for the database shall be carried out by anyone 
identifying a new risk or whoever is responsible for the risk. 

• Stone will serve as a general source of risk management expertise during this 
period. The facilitation team members will continue to provide facilitation on an as- 
needed basis. 


Section 3. Schedule of Activities 


[updated 2/1/96] 


Build 

Infrastructure 

Train Infrastructure 

Establish Baseline 

Tailor RM to IR- 
SIP 



Install Support 
Tools 



Train Project 
Personnel in RM 


Basic RM Practice 
Installation 

Improvement 

Cycle 

Evaluate for AA 



W/95 H/95 — ■ 12/ 9 5 2/96 2/96 4/96 — 5/96 
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3.1 Detailed Transition Schedule Milestones [added 11/16/95] 

The initial milestones for developing a risk management plan are 

• Document draft IR-SIP risk management plan (the tailored practice for IR-SIP): 
11/15/95 

• Final IR-SIP risk management plan: 1 1/20/95 

3.1.1 Basic Risk Management Practice Phase 

The basic risk management practice to be installed first includes the following: 

• all risk management activities at all levels of the IR-SIP project organization 

• database installed, tested, and all forms and templates to support the methods and 
tools incorporated 

The methods and tools to be used include everything but the mitigation status report 

and stoplight status report, which shall be held for later. 

• Although risks can be transferred to the AA Program Manager, there is no implied 
responsibility on the part of the AA Program Manager to provide data for the 
database. The IR-SIP PM shall assign the task of entering any risk data from the AA 
Program Manager. 

The detailed milestones for installing the basic practice are as follows: 

• Prototype risk database from SR & QA is installed and tested: G. Whitley: 11/18/95. 

• Tailored risk management training is developed by P. Stone and facilitation team: 
11/30/95. 

• All project personnel are trained on risk database and tailored risk management 
process: 12/15/95. 

• All top baseline risk areas have completed mitigation plans; plans are in place and in 
progress: 12/4/95. 

• Individual access to database for risk identification is available and is being used: 
12/1/95. 

• Weekly status meetings include risk as discussion topic using spreadsheet: 
12/18/95. 

• All risk information is being maintained in the risk database and risk information 
sheets are used as individual risk reports: 12/20/95. 

• New risks are being prioritized and action plans are being built: 12/30/95. 

• Progress Evaluation Points: 11/20/95, 12/20/95 


3.1.2 Improvement Phase 

The following will be implemented during the improvement cycle. 

• Monthly status meetings are using Stoplight Status Reports to indicate Top N risk 
status from IR-SIP PM to AA PM. 

• Mitigation Status Report is used for one of the top risks (provided its use is justified) 
by 3/1/96. 
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• Response time of database is improved by purchase of latest set of fixes from 
vendor. Need site license. Expected by 3/20/96. 

• Ability exists when printing risk spreadsheets to filter out risks not assigned to 
anyone in a specific work group. 

• New trending report is added to show average time required to close a Top N risk; 
average time Top N risk spends on watch list before final closing; average time to 
build mitigation plan; distribution of risks to responsible person (3/10/96). 

• AA viable procedure is tested for calculating actual mitigation costs against potential 
loss due to the risk (4/15/96). 

Progress evaluation points: 1/20/96, 2/20/96, 3/20/96, 4/20/96, 5/1/96. 

Based on evaluation, Stone and Johnstone present findings to other sponsors on 

4/30/96. Decision on whether or not to use risk management on the rest of the AA 

Program will be made at that point. 

Section 4. Allocated Budget and other Required Resources [updated 2/1/96] 

Funding is provided at the following levels: 

• SR & QA: $10,000 for tools and training, additional $3,000 for database 
upgrade and site license 

• AA Program: 0.5% of the program budget for FY96 

• IR-SIP: 1 % of the project budget 

Section 5. Evaluation Measures and Completion Criteria [updated 2/1/96] 

This risk management transition effort will be considered a success if the following 

outcomes have been met: 

1. An effective risk management practice is in place in the IR-SIP Project (document 
any major problems averted through management of risk in lessons learned part of 
risk database - collect for evaluation points as part of judging effectiveness of 
practice). 

2. AA Program management agrees to transition risk management to the rest of the 
AA Program. 

Measures to be used to evaluate the first outcome are 

• the number and severity of problems discovered late in the development lifecycle 
has decrease by at least 80% 

• 80% of project personnel and all managers find risk management has improved their 
ability to manage their tasks and make the right decisions 

• majority of project personnel do not find the practice to be unduly burdensome or 
inefficient 

• the estimated savings due to problems that were avoided is approximately 
equivalent to the resources invested in risk management by the IR-SIP project. 
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Section 6. Risks and Mitigation Strategies for this Implementation Effort [updated 

11/1/95] 

The following are the risks that the sponsors recognize as associated with this effort. 

Contingency or mitigation actions are also described. 

1. Too resource intensive: Resources used to perform risk management will be 
estimated and tracked. If resource usage exceeds 5% of personnel time on average 
with no visible benefit (in terms of significant problems avoided or reduced) by the 
first evaluation point, then the sponsors will revisit their decision to use risk 
management on this project. 

2. Ineffective basic risk management practice: If the tailored risk management practice 
designed for the project proves to need improvements or changes to more than 50% 
of it after two months of use, then the sponsors will revisit their decision and 
determine if a second attempt at tailoring the process is needed or if it is now too 
late to complete this effort with IR-SIP project. 

3. Unmotivated project personnel: The project personnel may find this too burdensome 
and not see the long-term benefits. Mitigation: Will brief the entire project early on to 
introduce the concept of risk management and demonstrate the sponsorship this 
effort has. Adjust project schedule, if needed, to allow for start-up time. Need to 
make sure people do not think more work is being piled on with no extra time to 
accomplish this. Sponsors/project manager need to stay alert to this issue. 

4. SR & QA database may not be useful. If it is not, the implementation schedule in 
this plan will slip by at least three weeks while we build an appropriate risk 
database. Testing on the SR & QA database will begin as soon as possible, using 
their equipment while waiting for the database to be installed on IR-SIP’s 
equipment. This should provide an answer on the database’s effectiveness a week 
sooner. 


Section 7. Establish Risk Baseline Method [updated 9/27/95] 

P. Stone has already been trained in conducting the Software Engineering Institute’s 
method for establishing a baseline set of risks and has trained the other members of IR- 
SIP’s facilitation team. The methods to be used include the following, taken from the 
Software Engineering Institute’s Continuous Risk Management Guidebook: 

• SEI Risk taxonomy-based interviews to be conducted with peer groups selected by 
Stone and Johnstone 

• Tri-level attribute evaluation 

• Classification by source using the taxonomy 

• Prioritization using multivoting 

• Planning the top three or four risk areas using problem-solving planning 

The Facilitation team will be led by Stone and will turn over all results to Johnstone, 
who will also report a summary of the results jointly with Stone to Goldman and R. 
Douglas (the other sponsors). 

Lessons learned from this baselining process will be used during the tailoring step to 
help tailor a more suitable process for these types of projects. Lessons learned will be 
documented by Stone and will be supplied to the sponsors. 
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Third - 

Establish a Risk Baseline 


ei 


Purpose: 

• generate a critical mass of project risks (motivation 
to manage risks) 

• begin the practice of Continuous Risk Management 

Description: A risk baseline should have 

• a list of risks (statements and contexts) 

• risk probability, impact, and timeframe 

• sets of related risks 

• prioritized risks based on project importance 

• plan of action for Top N risks/sets of risks 




You also need . . . 

Training and Project Familiarization 


Purpose: 

* ensure that members have information necessary 
to support the project roles 

* provide skills to use & implement the chosen tools 

* equip the team members with needed skills to 
establish a risk baseline 

* implement regular reviews of project risks r-™^— 

* establish a common vision of CRM ^' re 
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Act to - 

Install Basic Practice 

& 

Purpose: 

• install a basic set of activities that 


addresses all phases of the risk management 

paradigm 

• start simple and add complexity later 


Description: 

• involves establishing the basic set of risk 


management activities as defined in the 
implementation plan 

Rev 2. i/w 


Act to - 
Improve CRM 

& 


Purpose: 

• improve the basic Continuous Risk Management 
practice implemented during the Install phase 


Description: involves adding improvements 

• better match routine project management 
practice 

• increase efficiency of risk management activities 

• increase the forward-looking viewpoint 
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Hints and Tips 
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Summary -1 




Summary -2 

IR-SIP’s Risk Management Plan describes how IR-SIP 
will perform it’s tailored risk management process, 
methods, and tools 

• introduction 

* practice overview 

* project organization, roles, and responsibilities 

* practice details 

• risk management resources and milestones 

• risk information documentation 
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Summary -3 


Adapt risk management to your project. 

Document your practice and rationale. 

You will change and improve your risk 
management practice as you gain experience 
and learn from others’ experiences. 


Summary - Life-Cycle of a Risk 


Eventually risks go away 

• probability or impact goes to zero 

• risk becomes a problem 

Documenting the life cycle of risks 

• helps you learn what worked and didn’t work 

• should help you avoid similar difficulties 

• provides the opportunity to help other 
projects learn from your experience 
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Watch Out for 
Barriers - 1 


Resistance 

I don’t have the time. There’s too 
much regular project work to do. 


It’s not rewarded. Nobody wants 
to hear about what we can’t do. 


Answer 

If you don’t take the time now, you’ll take 
more time later to fix problems you could 
have prevented. 

Sponsors and management must be 
prepared to reward behavior they want to 


It’s a bureaucratic nightmare. The 
processes are too complicated and 
time consuming. 

I don’t want to look stupid, especially 
in front of upper management 

We already know our risks. We did an 
assessment at the beginning of the 
project Once is enough! 


It’s most successful when it’s tailored to 
the project management processes. 

Start simple and improve with time. 

Sponsors and managers should educate 
the project about what is expected. 

Has anything changed since you 
identified those risks? If so, then the risks 
are not the same. 


Watch Out for . . . 
Barriers - 2 


Resistance 

This is just another management 
initiative. I’ll wait to see if they’re 
serious before I put any effort into 
it. Why waste time and energy? 

They shoot the messenger. If I had a 
solution I wouldn’t need to bring it 
up in the first place. 


Answer 

It’s a valid question, but, if no one else 
improves, is that a valid reason for you 
not to improve? Don’t you want to be 
better than your competition? 

Sponsors and managers must 
encourage a risk-aware culture. Work 
with project personnel to identify 
potential solutions and choose one. 
Reward risk identification. 


Identifying risks means you need to 
solve them. We already have 
enough to do. 


Again, if you don’t take the time now, 
you’ll take more time later to fix the 
problems you could have prevented. 
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Module 10 


Course Summary 
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Overview 

Course objectives 


Definition of Risk 


Risk and Project Management 


Continuous Risk Management 


Risk Management Planning 


Risk Management Implementation 


Guidebook 
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Course Objectives 


Understand the concepts and principles of 
Continuous Risk Management and how to apply 
them 

Develop basic risk management skills for each 
function of Continuous Risk Management 

Be able to use key methods and tools 

Be able to tailor Continuous Risk Management 
to a project 


| NASA SATC 1 0-3 


Definitions of Risk 


Risk always involves the 

Risk should consider the 

likelihood that an 

severity of conseauence of 


undesired event will occur. 


the event should it occur 


x/: 

Quantitative Quantitative 


Risk = Likelihood * Severity 
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1 - Identify 


Purpose: 

• search for and locate risks before they become 
problems 

Description: 

• the process of transforming uncertainties and 
issues about a project into distinct (tangible) 
risks that can be described and measured 
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Risk Statement & Context 



Risk Statement 


A good risk statement 1 

• contains one condition 

• contains at least one consequence 

• is clear and concise 

Good context 

• provides additional information not in the risk 
statement 

• ensures that the original intent of the risk can 
be understood by other personnel, 
particularly after time has passed 
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2 - Analyze 
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3 - Plan 
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4 -Track 
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5 - Control 


Purpose: 

• make management decisions based on 
current information. 

Description: 

• the process that takes the tracking status 
reports for the project risks and decides what 
to do about risks based on the reported data 


[NASA SATC 10-15 
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Control Activities 



Evaluate - use tracking data to examine project 
risks for trends, deviations, anomalies, and 
identifying new risks. 


Decide - use tracking data to determine how to 
proceed with project risks 

- close 

- continue tracking and executing the 
current plan 

- replan 

- invoke a contingency plan 
Execute - implement control decisions 
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6 - Communicate & Document 


Purpose: 

* provide information and feedback to the 
project on the risk activities, current risks, 
and emerging risks 

Description: 

■ a process in which risk information is 
conveyed between all levels of a project team 
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Risk Management Planning - 1 












Risk Management Planning - 2 
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Translation Guide 

Description: The following tables provide a crossreference between terminology in the 
Continuous Risk Management Guidebook and the proposed risk management section 
4.3 of NHB 7120.5 Management of Major System Programs and Projects Handbook. 


Topic 

NHB 7120.5 (proposed) 

CRM Guidebook 

Definition of 
Risk 

A qualitative or quantitative probability that 
a program/project will experience 
undesired consequences such as failure 
to achieve a needed technological 
breakthrough, cost overrun, schedule 
slippage, or safety mishaps. 

Primary risk drivers are undesirable 
events whose probability is more likely 
than “remote” and whose consequences 
could pose a significant threat to mission 
success. 

Primary risk drivers typically fall into the 
following categories: 

• performance requirements and 
mission objectives 

• technology readiness 

• safety, reliability, maintainability, 
quality assurance, environmental 
protection 

• cost and schedule 

The possibility of suffering loss. In a 
development project, the loss describes 
the impact to the project, which could be 
in the form of diminished quality of the end 
product, increased costs, delayed 
completion, or failure. 

A statement of risk describes: 

• condition: the key circumstances, 
situations, etc., causing concern, 
doubt, anxiety, or uncertainty 

• consequence: the key, possible 
negative outcome(s) of the current 
conditions 

Risk 

Management 

Risk management covers the 
identification, assessment, mitigation, and 
disposition of risks at each stage of the life 
cycle. 

In particular, risk management begins with 
an identification of the general risk issues 
and concerns, based on program 
objectives and constraints. From these 
considerations, a plan is developed; 
followed by an assessment of specific 
risks. 

The Risk Management Paradigm 
illustrates a set of functions that are 
identified as continuous activities 
throughout the life of a project. 
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Topic 

NHB 7120.5 (proposed) 

CRM Guidebook 

Identifying, 

Assessing, 

and 

Mitigating 

Risks 

Risk assessment 

• identify primary risk drivers 

• estimate probability of occurrence 

• determine primary consequences 
given occurrence 

• assess cost and schedule impacts 

• mitigate technical, schedule, and cost 
risks 

Identify: 

• capture statement of risk 

• capture context of risk 

Analyze: 

• evaluate attributes (impact, 
probability, timeframe) of risks 
[qualitative or quantitative] 

• classify risks 

• prioritize (rank) risks 

Plan 

• assign responsibility (keep, delegate, 
transfer) 

• determine approach (research, 
accept, mitigate, watch) 

• define scope and actions 

Tracking and 

Controlling 

Risks 

A risk driver will be considered 
“controlled" or “retired" when any of the 
following conditions are satisfied: 

• risk mitigation options that reduce the 
probability of occurrence to “remote" 
have been planned and will be 
implemented 

• all reasonable mitigation options 
(within cost, schedule, and technical 
constraints) have been instituted, and 
all risk drivers determined to be more 
likely than “remote" have been judged 
by the appropriate PMC to be 
“accepted" 

• reserve funds are available so that, 
should the risk actually occur, 
resources would be available to 
recover from cost, schedule, and 
technical impact 

Track 

• acquire tracking data 

• compile tracking data 

• report tracking data 

Control 

• analyze status reports 

• decide how to proceed 

• execute decisions 

Considerations for closing a risk include 

• when the probability, impact, or risk 
exposure are either near zero or 
below an acceptable threshold as 
defined in the mitigation goal. The risk 
is considered to have been 
successfully mitigated. 

• when conditions have changed such 
that the risk is no longer relevant to 
the project 

• when a risk becomes a problem and 
must be tracked as such 
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Topic 

NHB 7120.5 (proposed) 

CRM Guidebook 

Risk 

Management 

Plan 

This plan guides the future risk disposition 

activity. This plan should include 

• risk management responsibilities, 
resources, schedules, and milestones 

• methodologies, processes, and tools 
to be used for risk identification, risk 
analysis, assessment, and mitigation 

• criteria for categorizing or ranking 
risks according to the probability and 
consequences; e g., risk matrix 

• role of risk management with respect 
to decision-making, formal reviews, 
and status reporting 

• documentation requirements for risk 
management products and actions 

This plan describes the risk management 
practice (processes, methods, and tools) 
to be used for a specific project. Contents 
include 

• introduction to plan and why it exists 

• overview of processes and how they 
relate to project management 

• the project’s involvement in carrying 
out risk management 

• details of each major activity and how 
it’s to be used 

• schedule, milestones, and resources 
required 

• how risk management information is 
documented, retained, controlled, and 
used 

Risk 

Information 

For each primary risk driver, the 

program/project should be prepared to 

present the following information 

• description of risk driver including 
primary causes and contributors to 
the risk 

• estimate of the probability (qualitative 
or quantitative) of occurrence, 
together with the uncertainty of the 
estimate 

• primary consequences should the 
undesirable event occur 

• significant cost impacts given its 
occurrence 

• significant schedule impacts given its 
occurrence 

• potential mitigation measures 

• implemented mitigation measures, if 
any 

• characterization of the risk driver as 
“acceptable" or “unacceptable” with 
rationale 

• 

Risk Information Sheet used to document 

information about a risk 

• unique identifier for risk 

• date risk was identified 

• statement of risk 

• context (associated information) that 
clarifies the risk 

• organization or person who identified 
the risk 

• priority ranking of the risk 

• likelihood of occurrence 

• degree of impact 

• timeframe in which the risk will occur 
or action is needed 

• classification of the risk 

• who is responsible for mitigating the 
risk 

• the selected strategy for mitigating the 
risk 

• a contingency plan, if one exists, and 
the event or time that triggers it 

• running status that provides a history 
of what is being done for the risk and 
changes to the risk 

• approval for mitigation strategies or 
closure 

• date when the risk was closed 

• rationale for closure of the risk 
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Software Risk Checklist 


The following is a software risk checklist. It is organized by development phases of a 
project, with emphasis on the software portion of the overall project lifecycle. Listed 
here are some , not an exhaustive list, of the generic risks that should be considered 
when any project contains software. This checklist contains practical questions that 
were gathered by experienced NASA engineers and is not a part of the SEI course or 
guidebook. The SEI has their own taxonomy-based questionnaire that should be 
considered during any risk assessment (SEI Continuous Risk Management Guidebook 
chapters A-32 to A-34, pg. 471-509). 

The project manager, software manager, system engineer/manager, any software 
technical leads, and the software engineers, as a minimum, should review, fill out, and 
discuss the results of this checklist. Taking into account all the different perspectives 
and adding risks specific to a project, the review team should then meet to create an 
agreed-upon set of risks and start planning how they will be addressed. This checklist 
is only an aid to start the managers and engineers thinking and planning how to realize, 
avoid, mitigate and accept the risks inherent in any software project. The first step to 
controlling a project is understanding where it may go out of control and plan to avoid it 
as much as is possible. As this risk checklist covers many lifecycle stages, it is 
suggested that this checklist initially be used during systems requirements to establish 
a baseline risk assessment. At that time, the entire risk checklist should be gone 
through and an initial risk assessment should be generated. These risks can then be 
documented in a risk database and/or a risk mitigation plan. Once this initial baseline 
risk assessment has been created, the project should revisit the risk checklist during 
each subsequent lifecycle stage in order to see if new risks have been discovered or if 
issues not previously understood to be a risk now need to be elevated to a risk. If the 
project is using rapid prototyping, the spiral lifecycle, or some other iterative lifecycle, 
then period at which the list will be revisited should be established at the beginning of 
the project and followed throughout. The software management plan or software risk 
management plan would be the appropriate place to document the entire risk approach, 
schedule and process. 

The checklist is laid out with the generic risks listed followed by a column to indicate if 
this is a risk for a particular project. Yes, this is a risk; No, not a risk for this project at 
this time; Partially a problem as stated, further clarification should be added. The last 
column is to indicate if this risk should be accepted or needs to be worked, i.e. the risk 
needs to be researched, mitigated, or watched. (See the SEI Continuous Risk 
Management Guidebook page 63.) 

Remember, this checklist is not an exhaustive list of all possible generic risks. It is 
meant to generate ideas and is not meant to be a complete list of all potential risks that 
could be considered. The user should consider the checklist, along with the Taxonomy 
Based Questionnaire provided in the SEI Continuous Risk Management Guidebook 
(Chapters A-32 to A-34, pages 471-509), as a basis for starting to examine possible 
risks on a project. The risk checklist should be added to, and tailored, to fit a 
project/program’s needs. Sometimes the wording on the questions contained in the 
checklist are open-ended in order to get the project team to think beyond what is 
written. 

Also remember, not all risks are technical. Development environment, schedule, 
resources, etc. all have risks that need to be considered. 
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System Requirements Phase 

"RISK 

Yes/No 

/Partial 


Are system-level requirements documented/ 

To what level? 

Are they clear, unambiguous, verifiable ? 



Is there a project-wide method tor dealing with tuture requirements 
changes? 


• 

Have software requirements been clearly delineated/allocated/ 



Have these system-level software requirements been reviewed, 
inspected with system engineers, hardware engineers, and the users to 
insure clarity and completeness? 



Have firmware and software been differentiated; who is in charge of 
what and is there good coordination if H/W is doing “F/W”? 



Are the effects on command latency and its ramifications on 
controllability known? 



is an impact analysis conducted for all changes to baseline 
requirements? 


' 
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Software Planning Phase 

RISK 

ACTION 

Is there clarity ot desired end product? 

Do the customer & builders (system and software) agree on what is to 
be built and what software’s role is? 



Are system-level requirements on software documented? Are they 
complete/sufficient and clearly understood? 



Are all interface requirements known & understood? 



Are roles and responsibilities tor system & software clearly defined 
and followed and sufficient? 



Have the end user/operator requirements been represented in the 
concept phase such that their requirements are flowed into the software 
requirements? 



Has all needed equipment, including spares, been laid out? and 
ordered? 

Is there sufficient lead time to get needed equipment? 

Is there a contingency plan for not getting all equipment? 

Is there a contingency plan for not getting all equipment when 
j needed? 


- 

Is the needed level ot technical expertise known? 



Is the level ot expertise tor sottware language, lifecycle, development 
methodology (Formal Methods, Object Oriented, etc.), equipment 
(new technology), etc. available: 
within NASA? 
from contractors? 

Will expertise be available as the schedule demands? 

Is there more than one person with a particular 
expertise/knowledge (i.e. is too much expertise held by only 
one team member? What if they quit, or get sick?) 



framing: 

Is there enough trained personnel? 

Is there enough time to train all personnel? 
on the project itself? 

on equipment/ software development environment, etc.? 
Will there be time and resources to train additional personnel as 
needed? 



Budget: 

Is the budget sufficient for: equipment? 

needed personnel? 

training? 

travel? 

etc. 
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Software Planning Phase (cont.) 

RISK 

ACTION 

Schedule: 

Is the schedule reasonable considering needed personnel, 
training, and equipment? 

Does the system-level schedule accommodate software 
lifecycle? 

Can needed equipment be made available in time? 



Has all the slack/contingency time on the critical path been used up/ 



Are software metrics kept and reported regularly/ Weekly/ Monthly/ 



Are deviations to the development plan being tracked? Trended / 
Are the trends reported in a manner to allow timely and appropriate 
software and project management decisions? 



Will new development techniques be used/ 



Will a new or different development environment be used/ 



Is this a new technology / 



Will simulators need to be designed and built/ 

Is there time and resources allocated for this? 



Is there a schedule that covers development of both ground and flight 
software? 

Is it reasonable, does it match reality? 

Is it being followed? 

Are changes tracked and the reasons for the changes well 
understood? 



Do the schedules tor ground and flight software match with what is 
needed for test and delivery? 



Are there separate schedules for flight and ground/ 
Are different people in charge of them? 

Are they coordinated by some method? 



Will test software need to be designed and developed/ 
Are there time and resources allocated for this? 



Distributed development environment: 

Will this be a distributed development (different groups or 
individuals working on parts of the project in different 
locations e.g. out of state)? 

Are there proper facilities and management structure to support 
distributed development? 



lnter/lntra group management: 

Are interfaces with other developers, suppliers, users, 
management, and the customer understood and documented? 
Is there a known way to resolve differences between these 
groups (i.e., conflict resolution/ who has ultimate authority, 
who is willing to make a decision)? 
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Software Planning Phase (cont.) 

RISK 

ACTION 

Management Planning: 

Is management experienced at managing this size and/or type 
of team? (Is there an experienced project manager?) 

Is management familiar with the technology being used (e.g.. 
Formal Methods, OOA/OOD and C++)? 

Is there a well-constructed software management plan that 
outlines procedures, deliverables, risk, lifecycle, budget, etc. 
Is it reasonable, does it match reality? 

Is it being followed? 



Does software lifecycle approach & timeframe meet needs of overall 
project; does it have a chance of being close to what is needed? 



Has time been allotted for safety analysis and input? 



Has time been allocated tor reliability analysis (e.g., failure Modes 
and Effects Analysis (FMEA), Critical Items List (CIL), Fault 
Tolerance Analysis) input? 



Has time been allocated tor software (s/w) quality analysis input and 
auditing? 



Have software development standards & processes been chosen/ 



Have software documentation standards been chosen/ 



Has Software Product Assurance given input on all standards, 
procedures, guidelines, and processes? 



Is funding likely to change from originally projected? 

Is there a plan in place to handle possible funding changes? 
Prioritization of requirements? 

Phasing of requirements delivery? 



Is there a procedure/process tor handling changes in requirements/ 
Is it sufficient? 



Examine detailed technical considerations such as: 

Can the bus bandwidth support projected data packet transfers? 

Are system requirements defined for loss of power? 

Is the system reaction to loss of power to the computers 
known or planned for? 

Have UPS (Uninterruptable Power Supplies) been planned for 
critical components? 
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Software Requirements Phase 

RISK 

ACTION 

Software schedule: 

Is there an adequate software schedule in place? 

Is it being followed? 

Are changes to schedule being tracked? 

Are changes to the schedule made according to a planned 
process? 

As events change the schedule, is the decision process for 
updating the schedule also examined? That is, question if there 
is something wrong in the process or program that needs to 
change in order to either make schedule or affect the schedule- 
updating process? 

Has the overall schedule been chosen to meet the needs of true 
software development for this project or has the software 
schedule merely been worked backwards from a systems 
NEED date with no consideration for implementation of 
recommended software development process needs? 



Has all the slack/contingency time on the critical path been used up? 



Are software metrics kept and reported regularly? Weekly? Monthly? 



Are deviations to the development plan being tracked? 1 rended? 
Are the trends reported in a manner to allow timely and appropriate 
software and project management decisions? 



Are parent documents baselined before child documents are reviewed? 
Is there a process in place for assessing the impact of changes 
to parent documents on child documents? 

Is there a process in place for assessing the impact of changes 
to parent documents from changes within child documents? 



Are review/inspection activities and schedules well defined and 
coordinated with sufficient lead time for reviewers to review material 
prior to reviews/inspections? 



Is there a process tor closing out all 1 BDs (to be determined) before 
their uncertainty can adversely affect the progress of the project? 



Have all the software-related requirements from the systems-level 
requirements been flowed down? 

Have the system level and software level standards been chosen? 
Have the requirements from these standards been flowed down from 
the system level? 

Have guidelines, etc., been established? 
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Software Requirements Phase (cont.) 

RISK 

ACTION 

Has the project planned how to handle changing requirements? 
Compartmentalized design? 

Are procedures/change boards in place for accepting/rejecting 
proposed changes 

Are procedures in place for dealing with schedule impacts due 
to changes? 

Is the project following these procedures? 

Is there good communication with the principle 
investigators/customer? 

Have requirements been prioritized? 

Is this prioritization tracked, reviewed, and periodically 
updated? 

Is there a clear understanding of what is really necessary for 
this project? 



Have there been changes/reductions m personnel smce first estimates? 



Are there sufficient trained software personnel? 

Does all the knowledge for any aspect of project reside in just one 
individual? 



Is there a software testmg/venfication plan? 



Is the software management plan being followed? 
Does it need to be adjusted? 



is the software development environment chosen and m place? 



Does work contracted out have sufficient controls and detail to assure 
quality, schedule, and meeting of requirements? 



Is a Software Configuration Management (SCM) Flan m place and 
working? 



Are backups of SCM system/database planned and earned out on a 
regular basis? 



Are inspections or peer reviews scheduled and taking place? 



Software Quality/Product Assurance (SQA or SFA): 

Is SPA working with development to incorporate safety, 
reliability and QA requirements? 

Is s/w development working with SPA to help establish 
software processes? 

Does SPA have a software-auditing process and plan in place? 



Are there good lines of communication established and working 
between software project groups? 
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Software Requirements Phase (cont.) 

RISK 

ACTION 

Are good lines ot communication established and working with groups 
outside software development? 

Are there written agreements on how to communicate? 

Are they followed? 

Are they supported by management and systems group? 

Are there good interface documents detailing what is expected? 
Did all the concerned parties have a chance to review and agree 
to them? 



Have resources been re-evaluated (equipment, personnel, training, 

etc.)? 

Are they still sufficient? 

If not, are steps being taken to adjust project schedule, budget, 
deliverables, etc. (more personnel, re-prioritization and 
reduction of requirements, order new equipment, follow 
previously established mitigation plan, etc.)? 



Are (JO 1 being used? 

How are COTS maintained? Who owns and who updates 
them? 

Is the product affected by changes to COTS? 

Will new releases of one or more COTS be 
maintained/supported? 

Are COTS releases coordinated with the developed software 
maintenance and releases? 

Do COTS meet the necessary delivery schedule? 

Do personnel have a good understanding of how to 
use/integrate COTS into final product? 

If the COTS incorporated into the system meet only a subset of 
requirements of the overall requirements (that is, the COTS 
software does not completely fulfill the system requirements) , 
have the integration task and time been correctly estimated for 
merging the COTS with any in-house or contracted software 
that is needed to complete the requirements? Can this 
integration task be estimated? 

Will custom software need to be written to either get different 
COTS to interact correctly or to interact with the rest of the 
system as built or planned? 

; 

i 

i 

i 

1 


Is a new technology/methodology being incorporated into software 
development? Analysis? Design? Implementation? (e.g.. Formal 
Methods. Object Oriented Requirements Analysis, etc.) 

Has the impact on schedule, budget, training, personnel, 
current processes been assessed and weighed? 

Is there process change management in place? 




Software Requirements Phase (cont.) 

RISK 

ACTION 

Is a new technology being considered for the system? 

Has the impact on schedule, budget, training, personnel, 
current processes been assessed and weighed? 

Is there process change management in place? 



Is the project planning to do prototyping of unknown/uncertain areas 
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to find out it there are additional requirements, equipment, and/or 
design criteria that may not be able to be met. 
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Software Design Phase 

RISK 

ACTION 

Is the software management plan being followed.'’ 
Does it need to be updated? 



Is the requirements tlow-down well understood/ 



Are standards and guidelines sufficient to produce clear, consistent 
design and code? 



Will there be, has there been, a major loss ot personnel (or loss or 
critical personnel)? 



Is communication between systems and other groups (avionics, tlmds, 
operations, ground software, testing, QA, etc.) and software working 
well in both directions? 



Requirements: 

Have they been baselined 
& are they configuration managed? 

Is it known who is in charge of them? 

Is there a clear, traced, managed way to implement changes to 
the requirements? (i.e., is there a mechanism for inputting new 
requirements, or for altering old requirements, in place and 
working)? 

Is there sufficient communication between those creating & 
maintaining requirements and those designing to them? 

Is there a traceability matrix between requirements and design? 
Does that traceability matrix show the link from requirements 
to design and then to the appropriate test procedures? 



Has bystem batety assessed software / 

Does any software involved hazard reports? 

Does software have the s/w subsystem hazard analysis? 

Do software personnel know how to address safety-critical 
functions, how to design to mitigate safety risk? 

Are there fault detection, isolation, and recovery (FDIR) 

■ techniques designed for critical software functions? 



Has software reliability been designed tor/ 

What level of fault tolerance has been built in to various 
portions /functions of software? 



Is there a need to create simulators to test software / 

Were these simulators planned for in the schedule? 

Are there sufficient resources to create, verify and run them? 
How heavily does software completion rely on simulators? 
How valid/accurate (close to the flight unit) are the simulators? 
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Software Design Phase (cont.) 

RISK 

ACTION 

Are simulators kept up-to-date with changing llight H/W7 



How heavily does hardware completion rely on simulators? 



Is firmware and/or any other software developed outside the software 
flight group ? 

Is it being integrated? 

Is it being kept current based on changes to requirements & 
design? 

Is it configuration managed? 



Does work contracted out have sufficient controls and detail to assure 
quality, schedule, and meeting of requirements? 



Will design interfaces match m-house or other contracted work? 



Is a software configuration management plan in place and working? 



Are backups of SCM system/database planned and earned out on a 
regular basis? 



Are Inspections and/or peer reviews scheduled and taking place? 



Software Quality /Product Assurance (SQA or SPA): 

Is SPA working with development to incorporate safety, 
reliability, and QA requirements into design? 

Does SPA have a software-auditing process and plan in place? 
Have they been using it? 



Are parent documents baselined before child documents are reviewed? 
Is there a process in place for assessing the impact of changes 
to parent documents on child documents? 

Is there a process in place for assessing the impact of changes 
to parent documents from changes within child documents? 



Are review/inspection activities and schedules well defined and 
coordinated with sufficient lead time for reviewers to review material 
prior to reviews/inspections? 



Has all the slack/contingency time on the critical path been used up? 



Are software metrics kept and reported regularly? Weekly? Monthly? 



Are deviations to the development plan being tracked? t rended? 
Are the trends reported in a manner to allow timely and appropriate 
software and project management decisions? 




A2-11 




Continuous Risk Management 


Software Implementation Phase 

RISK 

ACTION 

Coding and unit test 



is the software management plan still being used/ 
Is it up-to-date? 



Are there coding standards / 



Are they being used? 



Are software development lolders (SDhs) being used to capture design 
and implementation ideas as well as unit test procedures & results? 



Are code walk-throughs and/or mspections being used.' 
Are they effective as implemented? 



Is SOA/SPA auditing development processes and SDl-s/ 



Is the design well understood and documented / 



Are requirements being tlowed down through design properly/ 



Is the schedule being maintained/ 

Have impacts been accounted for (technical, resources, etc.)? 
Is it still reasonable? 



Has all the slack/contingency tune on the critical path been used up/ 



Are software metrics kept and reported regularly / Weekly'/ Monthly / 



Are deviations to the development plan being tracked/ trended/ 
Are the trends reported in a manner to allow timely and appropriate 
software and project management decisions? 



Have any coding requirements tor salety-cntical code been 
established? 

If so, are they being used? 



Does the chosen development environment meet llight 
standards/needs? 



Has System Safety assessed software (subsystem satety analysis// 
Has software reviewed this safety assessment? 

Has software had input to this safety assessment? 

Do software personnel known how to address safety critical 

functions? , 

Is software working with systems to find the best solution to 

any hazards? 



Has FDlk (tault detection, isolation, and recovery) and/or fault 
tolerance been left up to implemented (i.e., no hard requirements 
and/or no design for these)? 



Is there a known recourse/procedure lor design changes / 

Is it understood? 

Is it used? 

Does it take into account changes to parent documents? 
Does it take into account subsequent changes to child 
documents? 
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Software Implementation Phase (cont.) 

RISK 

ACTION 

Coding and unit test (cont.) 



Is there a known recourse/procedure tor requirements changes'.^ 
Is it understood ? 

Is it used? 

Is it adequate; does it need to be altered? 

Does it take into account changes to parent documents? 
Does it take into account subsequent changes to child 
documents? 



Is there development level Software Configuration Management 
(SCM) (for tracking unbaselined changes and progress)? 

Is it being used by all developers, regularly? 

Are backups performed automatically on a regular basis? 



Is there formal SCM and baselining of requirements and design 
changes? 



Are the design documents baselined? 


- 

Are the requirements baselined? 



Have test procedures been written and approved? 

Are they of sufficient detail? 

Will these tests be used for acceptance testing of the system? 
Are these procedures under SCM? 

Are they baselined? 



Do some software requirements need to be tested at the systems level 
for complete verification? 

Are these documented? 

Do the systems-level test procedures adequately cover these? 
Does the requirements/verification matrix indicate which 
requirements are tested at the systems level? 



For subsystem-level testing: 

Has software been officially accepted by the subsystems (sign- 
off, baselined)? 

Are software testing facilities maintained for any regression 
testing? 



Are unit testing procedures and results maintained via SCM/ 



Is there auto-generated code'/ 



Is unit testing planned tor auto-generated code? 

Are there procedures for testing unit level auto-generated code? 



Are implementation personnel familiar with the development 
environment, language, and tools? 

Sufficiently trained coders (e.g., understand OOA, OOD, C++, 
Formal Methods, etc., whatever is needed)? 

Sufficient level of expertise (not first or second time ever done, 
not just trained)? 
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Software Implementation Phase (cont.) 

RISK 

ACTION 

Codins and unit test (cont) 



Are coders sufficiently laminar with project tunction/design? 



Do coders have ready access to someone with sufficient expertise and 
whose time is available for participation in code walk-throughs or 
inspections and for technical questions? 



Is there sutticient equipment/ 



Are there build procedures? 

Are they documented? 
Are they under SCM? 
Are they being followed? 



Are there bum procedures tor any PKUMS / KUMS / fcFKUMS / 

Are they documented? 

Are they under SCM? 

Are they being followed? 

Do they include a method for clearing PROMs (if applicable) 
and checking them for defects prior to burning? 

Does the procedure include a method to determine and 
recording the checksum(s)? 

' 

- 

Are test plans complete? 

Is further testing needed? 

Unit level testing? 

CSCI level testing? 
Integration testing CSCIs? 
System-level testing? 



Is the test/requirements matrix up to date/ 
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Software Implementation Phase (cont.) 

RISK 

ACTION 

Integration and Systems 'testing 



Are review activities and schedules well detined and coordinated? 



Is there a sufficient number of experienced test personnel? 

Who are experienced on similar projects? 

Who are experienced with this project? 

Who are experienced with test equipment, set-up, simulators, 
hardware? 

Who are experienced with development environment? 



Is the software test plan being lollowed'. r 
Does it need to be modified? 

Does it include COTS? 

Does it include auto-generated code? 



Are there well-written, comprehensive test procedures? 
Are they up to date? 

Do they indicate the pass/fail criteria? 

Do they indicate level of regression testing? 



Are test reports written at the time of the tests? 



Are test reports witnessed and signed off by SfA? 



Is the test/requirements matrix up to date? 



Is there a known recourse/procedure tor testing procedure changes? 
(i.e., is there an Software Configuration Management Process that 
covers the test procedures?) 

Is it understood? 

Is it used? 

Does it take into account possible changes to parent documents 
of the test plan or other parent documents? 

Does it take into account subsequent changes to child 
documents? 

Does it take into account regression testing? 



Is there a known recourse/procedure for requirements changes? 

Is it understood? 

Is it used? 

Is it adequate, does it need to be altered? 

Does it take into account changes to parent documents (e.g., 
systems requirements)? 

Does it take into account subsequent changes to child 
documents (e.g., design and testing documents)? 



Is there Software Configuration Management (SCM) (tor tracking 
baselined changes and progress)? 

Is it being used? 

Are backups performed automatically on a regular basis? 
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Software Implementation Phase (cont.) 

RISK 

ACTION 

Integration and Systems I esting (cont.) 1 



Is there iormal SCM and baselining ot requirements and design 
changes? 



Are the design documents lormally baselined and in SCM / 



Are the software requirements formally baselined ! 



Have test procedures been wntten and approved/ 

Are they of sufficient detail? 

Do they exist for unit test? 

Do they exist for CSCI level testing 

Do they exist for CSCI integration-level testing? 

Do they exist for software system-level testing? 

Will these tests be used for acceptance testing to the system? 
Are these procedures in SCM? 

Are they baselined? 



Do some software requirements need to be tested at the systems level 
for complete verification? 

Are these requirements verification procedures documented? 
Where are they documented? In software test procedures? In 
systems test procedures? 

Do the systems-level test procedures adequately cover these? 
Does the requirements/verification matrix indicate which 
requirements are tested at the systems level? 



f or system-level testing: , 

Has software been officially accepted by systems (sign-off, 
baselined)? 

Are software testing facilities maintained for any regression 
testing? 



Is firmware ready and tested? 
Is it baselined and in SCM? 



Are tEere separate test personnel that have not been designers or coders 
scheduled to perform the tests? 

Do they need training? 

Is time allowed for their unfamiliarity with the system? 



On the flip side, are testers too familiar with software/ Will they have 
a tendency to brush over problems or fix problems without going 
through proper channels/procedures? 

j 


Have requirements/design/code personnel been moved to other tasks 
and are no longer available to support testing or error correction? 



Are test pass/tail criteria known and understood / 
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Software Implementation Phase (cont.) 

RISK 

ACTION 

Integration and Systems t esting (cont.) 



Is regression testing planned tor? 

Is there time in the schedule for it? 

Have estimates been made at each test point of the amount of 
regression testing necessary to cover fixes if test fails? (e.g., certain 
failures require complete (end-to-end) re-testing, others may require 
only re-testing of that test point.) 



Is ground software (or other related software) available tor testing or 
for use in testing flight s/w? 



Has testing ot COTS at the software system level been adequately 
covered and documented? 

Are there test procedures specifically for proving integration of 
COTS? 

Does the requirements to test matrix indicate where COTS is 
involved? 



Has testing ot COTS at the system level been adequately covered and 
documented? 



Is there good configuration management in place? 

Is it used? 

Is there version control? 

Is error/failure tracking in place? 

Are PRACA (Problem Report and Corrective Action) and/or 
s/w change records created? 

Are problem/change records tracked to closure? 

Is error correction written into each new release of a module (in 
code comments, in file header, in SCM version description)? 
Are incorporated PRACAs listed in the build release version 
descriptions? 



Will a tight schedule cause: 

Dropping some tests? 

Incomplete regression testing? 

Dropping some fixes? 

Insufficient time to address major (or minor) design and/or 
requirements changes? 

No end-to-end testing? 

Are these issues being addressed? 

Who makes these decisions? The change control board? 
How are they recorded? 

Does the version description document (VDD) indicate true 
state of delivered software? 
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Software Implementation Phase (cont.) 

RISK 

ACTION 

Integration and Systems 1 esting (cont.) 



Has all the slack/contingency time on the critical path been used up/ 



Are software metrics kept and reported regularly/ Weekly / Monthly / 



Are deviations to the development plan being tracked/ trended/ 
Are the trends reported in a manner to allow timely and appropriate 
software and project management decisions? 




A2-18 



Continuous Risk Management 


Acceptance Testing and Release 

RISK 

ACTION 

Has pre-ship review already taken place'/ 



Is actual flight equipment available lor software testing? 

Do the logbook and test procedures record actual flight 
hardware used for testing? 



Are pass/lail criteria established and tollowed? 



Is a regression testing procedure documented and known? 
Is it used? 



Is the procedure to handle P RAC As (Problem Report and Corrective 
Action) at the acceptance level documented? 

Is there a change review board in place? 

Has there been configuration management of changes? 

Is the PRACA/SPCR (S/W Problem and Change Request) log 
maintained with status? 



Is systems-level testing adequate to insure software requirements 
or some software-level testing done separately and documented? 



Is appropriate personnel witness and sign-ott testing? 
SPA or QA involved? 



Are all parts ot the architecture verified on the ground prior to flight? 



Does a complete VDD (Version Description Document) exist? 

In the VDD, are: 

All delivered software release versions listed? 

All COTS and their versions listed? 

All hardware versions appropriate for this release noted? 
SCM release description(s) provided? 

Build procedures given? 

Bum procedures given? 

Installation procedures provided? 

List of all incorporated (closed) problem reports and change 
requests included? 

List of all outstanding problem reports and change requests 
included? 

List of any known bugs and the work-arounds provided? 
Changes since last formal release indicated? 

List of all documentation that applies to this release, and its 
correct version, provided? 

If there are known discrepancies to hardware, documentation, etc. are 
these listed and discussed in the VDD? 



Is there clean customer hand-ott: 

Up to date documentation? 
User/Operations Manual? 

Code Configuration Managed? 
All PRACAs & SPCRs closed? 
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Acceptance Testing and Release (cont.) 

RISK 

ACTION 

Is there good conliguration management wrap-up: \ 

Is there a method for future updates/changes in place? 

Proper off-site storage of data, software and documentation? 
What happens to SCM and data when project is over? 
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APPENDIX C: RELIABILITY DESIGN CHECKLIST 


Example taken from: 

Reliability (FI) and Maintainability (M) 
Design Checklist 
NAVSEA S0300-AC-MMA-01 O-R&M 

October 1 977 


Obtainable from: 

Naval Publications and Forms Center 
5801 Tabor Ave 

Philadelphia, Pennsylvania 19120 
Attn: Code F01G 
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R/M PROGRAM ELEMENTS 

PROGRAM PLAN 

ORGANIZATION 

SUBCONTRACTOR & 
SUPPLIER CONTROL 

PROGRAM REVIEW 

R ANALYSIS 
MODEL 

THERMAL ANALYSIS 

ALLOCATION 

PREDICTION 

SIMILARITY 
AVERAGE STRESS 
DETAILED STRESS 

PART CONTROL 

FMSEA/FAULT TREE 

CRITICAL ITEM CONTROL 


PRODUCTION FOLLOW ON 
TYPE 01 

NEW 

DEVELOPMENT 

ABC 

XXX 

XXX 

X 


X 

X X 
X 

X 

X 

X 

X X 
X X 
X X 


STORAGE EFFECTS X 

DESIGN REVIEW X X 


CONTRACT 

MODIFIED 

DEVELOPMENT 

ABC 

XXX 

XXX 

X 


X 

X X 
X 

X 

X 

X 

X X 
X X 
X X 
X 

X X 


NOTE: See next page for explanation of A, B, and C, above. 
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RSM LEVELS 

LEVEL A 

• HIGH LEVEL OF SAFETY 

• CRITICAL SYSTEM 

• AND N £XPENS?VE ICAL/ MAINTENANCE 01 FFICULT 

LEVEL B 

• SAFETY FACTOR IN DESIGN 

• MODERATELY CRITICAL SYSTEM 

• MAINTENANCE MODERATELY DIFFICULT AND EXPENSIVE 

L EVEL C 

• SAFETY OF MINIMUM CONCERN 

• LOW SYSTEM CRITICALITY 

• DOWNTIME NOT CRITICAL 
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No. 

21 


(a) 

(b) 

(c) 

<<*) 


<«) 

if) 

<0 

(b> 


(i) 


0) 

(*> 

fl) 

(m) 


RELIABILITY (R) DESIGN CHECKLIST 


Item Description 
Management 

Does contractor have a permanent in -bouse R 
staff? 

Is staff composed of experienced R, engineers? 

Does program R^ engineer report directly to pro- 
gram manager? 

Does R group have the facility/ authority to in- 
terface directly with other engineering groups; 

(1) Design? 

(2) Systems engineering? 

(3) Quality Control? 

(4) Integrated Logistics support? 

(5) Procurement? 

(61 Test and Evaluation? 

Is R group representatives) member (s) of 
design review team? 

Does R group review all drawings and specifica- 
tions for adequacy of R. requirements ? 

Does R program engineer have sign-off authority 
on all drawings and specifications ? 

Does R englnee^/group review Purchase Orders 
and Purchase specifications to assure all parts 
and subasse m blies are procured with adequate R 
requirements ? 

Does R group have membership and a voice in 
decisions for the following: 

(1) Material Review Board? 

(2) Failure Review Board? 

(3) Engineering Change Review Board? 

Is R group represented on surveys and quality 
audits of potential subcontractors? 

Is R group represented at subcontractor design re- 
views and meetings where R is a topic of discussion? 

Does an R, group member(s) monitor/witness sub- 
contractor R tests ? 

Does R group contain experts in the fields of com- 
ponents /failure analyses? 


■Yes No 


Remarks 
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No. 


22 


(*> 

(b) 

ic) 

id) 

(*> 

it) 

it) 

<b> 


(1) 

a) 


i*> 

0) 


(m) 


(n) 

(o> 

IP) 


w 

<r) 


IS) 


tt ) 
Cul 


Hem Description 


RELIABILITY (Rj DESIGN CHECKLIST 
Yea No 


Design for R 

THERMAL REQUIREMENTS: 

Have deLaiiod thermal analysis been performed to 
determine component/ module ambient operating 
tempera cure ? 

Has a unit similar to final configuration (e.g. . 
brass board. preproduction unit. etc.), been instru- 
mented to develop a thermal mapping of the design? 

Have anemometer probes been used to measure 
coolant air flow patterns ? 

Are equipment internal cooling considerations o 
sufficient to limit internal temperature rises to 20 C 
maximum ? 

Are high power dissipation components (e.g.. large 
power resistors, diodes, transformers, etc.) heat 
staked? 

Where chilled water or chilled air is used for 
cooling have hermetically sealed components been 
selected due to possible moisture condensation? 

Where chilled water or chilled air is used for 
cooling are components shielded or otherwise pro- 
tected from moisture condensation? 

Where chilled water or chilled air is used (or 
cooling has consideration been given to removal of 
condensation to avoid accumulation of moisture and 
possible fungus growth or corrosion within the 
equipment? 

Are ail printed circuit boards conformally coated? 


Have circuit performance tests been conducted at 
high and low temperature extremes to assure circuit 
stability over the required operating temperature 
range ? 

Do heat conducting surfaces make good contact {no 
air gaps) and have low thermal resistances? 

Do surface coatings and paints provide good con- 
duction, convection and radiation coefficients for 
heat transfer 

Do adhesives wnere used for fastening components 
to ?CB*s or chassis nave good thermal conduc- 
tive properties'* ^ 

Do potting . encapsulation and conform ai coating materials 
wnere used have good thermal conducting properties? 

Have differences m thermal expansion of inter- 
facing materials been taken into account'* _ 

Are hign power dissipation components mounted 
directly to me cnasis for better heat sinking rather 
than encapsuiaiec or thermally insulated'* 

Is thermal contact area between components and 

neat sinks *epi to a maximum'' _ 

Are components sensitive to nest located away from 
neat flow pains. power supplies and otner oign power 
dissipation components ** 

Are air gaos or tnermal insulation provided where 
necessary ;o avoid neat flow to temperature sensi- 
tive components “* _ 

Are temperature overload devices, alarms used to 
prevent damage due ui loss oe cooling apparatus ** _ 


Do inlet temoerature ducts have filters to prevent 
accumulation of dirt on sssemohes wmch would result 
in reduction of neat transfer'* 

Do components mounted oo ?C3’s nave lOcquaie lead 
lengtrva xnc *re '.ne tormed tw relieve ieuU 

stresses during '.nermal expansion and contraction ** 
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NO. 


Horn Description 


RELIABILITY (RJ design checklist 

Ym No 


Ac 


■/ 

•/ 


<w) 

W 

c y) 

U) 

(w>) 

(CC) 

(da) 

<•«) 

(XT) 

Ibh) 

(li> 

(in 


VIBRATION/SHOCK/STRUCTURAE REQUIREMENTS: 


Hu analysis boon performed to determine resooem 
frequencies to oe experienced is t be equipment 
environment ? 

Have detailed vtbratioo/ehock/ structural analyse* 
been performed to validate structural Integrity of 
the design? 

Have critical /unique assemblies boon instrumented 
with accelerometers and tested to verily design ade- 
quacy with respect to vibration and abode sranamiaei- 
bility factors? 

Have structural mountings been designed to r eson at e 
away from resonant frequencies and their harmonica? 

Have damping considerations been applied to sub- 
assemblies and components mounting where natural 
frequencies are close to expected environmental 
frequencies ? 

Are large .components (over 1/2 ox. ) being damped 
or tied down to the chassis or printed circuit boards 
to prevent high stresses or fatigue failure of elec- 
trical leads? 

Heavy components are mounted near corners of the 
chassis near mounting points for direct structural 
support rather than be t wee n supports ? 

Centers of gravity of heavy components are kept low 
dose to tbs plane of tbs mounts ? 

Are cables /harnesses damped close to terminal 
connections to avoid resonances and prevent stress 
sno failure at the point of connection? 

Do eables/wires have sufficient slack to prevent 
stresses during thermal changes and mechanical 
vtbrmti on/ shock ? 

Stranded wire u used wben cabling might be suscep- 
tible to fatigue failure? 

Components and suoansemoi&ee have adequate sway 
space to avoid collision curing vibration and shock? 

Welding (not spot welding) and/or riveting is used 
for permanently attached structural members rather 
than nuts and bolts ? 

All comooneni leads have minimum bend radii to 
avoid overstressing n 
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No. 


(tic) 

( a > 


RELIABILITY (R> DESIGN CHECKLIST 
Hem Description y e# No 

MISCELLANEOUS REQUIREMENTS: 

consideration been given to avoid the use of 
disucniiar metais ° 

Have the PCB's been design*! for the following 
consideration*: 

(1) PCB material :* compatible with storage and 
operating temperature (plus operating tempera- 
ture rises) with respect to: 

(1) PCB material? 

(2) Metal cladding/bonding strengths? “ ' 

(3) Board warping? 


(mm) 


(nn> 


too) 

<PP) 


( 2 ) 


(3) 

(4) 

(5) 


PCB resistivity is sufficiently high to meet cir- 
cuit leakage current requirements even under 
high humidity ? 

PCB arc resistance is sufficiently high where 
high voltages are present? 

PCB dielectric constraint is sufficiently low to 
prevent building up of unwanted capacitances ? 
PCB flexural strengths (function of board 
material and dimensions) is sufficient to meet 
structural and vibration requirements? 

PCB conductors width is sufficient to handle 
maximum current flow without harmful beat 
generation or resistance drop’ 

PCB's have plated through holes to aid in 
soldering of lead electrical connections ’ 

PCB conductor spacing* have a minimum 
spacing based upon voltage between conductor 
(a* g. , .025" per 150 volts peak)? 

PCB conductor paths are spaced and designed 
to keep capacitance between conductors to a 
minimum ? 

Are PCB l s conformally coaled? 


( 1 ) 

( 2 ) 

( 3 ) 

(4) 

(5) 
f€) 


Good thermal conductivity for heal transfer* 7 
Good electrical iaol an on/di electric ? 
Provide dampening for shock and vibration? 
Thermal expansion coefficient* watch match 
those of items encapsulated? 

Win not crack or shatter under vibration and 
mechanical and thermal shock? 

Has good chemical stability under anticipated 
use environments ? 


Have worst case analyse, or statistical variation 
of parameters oeen conducted to determine required 
component electrical tolerances considering- 

ill Mlnu/arniviBir • 


( 1 ) 
1 2 ) 

(3) 

(4) 

(5) 


Manufacturing tolerances ‘ 

Tolerances due to temperature changes? 

Tolerance* due to agings 
Tolerance* due to humidity? 

“ h ' gn •' r “ ,Uency or other operzung 


C ° Qa,der * f ° r CrUlcaJ *««*»« 

in ven e r< * 1 ' u *** ac y l ® J ®«1. Has consideration* been 
f lvo,a cornmon mode (aUure s.tuanoo# wnich 
could diaapie ail redundant circuits' 7 


Remarks 
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No. 


RELIABILITY (R) DESIGN CHECKLIST 
Item Description Yea_ No 


Remaj 


(<W) 


(rr) 


(*•) 


Has design practices been applied to obtain RFI 

suppression such as: 

(1) Use alternating current ooo-commutaring machin- 
ery rather than direct current machinery when 
feasible? 

(2) Provide optimum interference suppression with 

two twisted wires in a common shield whenever wire 
pairs can be used? 

(3) Use short wires in preference to long wires? 

(4) Filter power lines to remove harmonica and 
other types of inherent interference? 

(5) Mount filters as close to interference sources 
as possible without altering the effectiveness of 
the filter? 

(6) Use bonding techniques to insure that good elec- 
trical contact is made between chassis, conduit, 
shielding, connectors, structural and hous i ng 
metal parts? 


(?) Remove non-conducting costings from bolts, 
nuts, and tapped holes? 

(8) Internally shield iavididual sections of equipment 
which are either highly susceptible to inter- 
ference or which general* Interference. For 
example, the r-f input stages and local oscillators 
should be shielded individually? 

(9) Use a bandwidth consistent with the minimum 
possible value for the received signal. This often 
improves the signal-to-ooise ratio? 

(10) Use direct current filament sources where 
practicable ? 

(11) Ground center tap of filament transformer 
secondary winding to reduce hum ? 

(12) Avoid the use of gaseous lighting devices in the 
vicinity of sensitive wiring or electronic 
equipment? 

(13) Do not cable noisy and clean leads together? 

(14) Never route cables near knows interference 
sources? 

(15) Do not use shields or metal structures for return 
current paths? 

(1G) Avoid the use of corrosion preventive compounds 
with high insulating qualities at bond joints? 

Have considerations been given to preclude damage 

due tor 

( 1 ) installation ? 

(2) Handling? 

(3) Transportation? 

(4) Storage? 

(5) Shelf Life ? 

(6) Packaging ? 

(7) Maintenance environment? 


(8) Other environments: 

(a) Humidity'* 

(b> Fungus' 1 

(c) Sand and dust? 

id) Sait atmospnere' 1 

Has reliability oeen considered as a factor ;n all 
tradeoff studies affecting equipment reliability'* 
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No. 


Item Description 


23 Parts Program 


RELIABILITY (R) design checklist 
Y es No 


la) 


a» 

(C) 

(d) 

<«) 

in 

(Z) 

(b) 

(i) 


0 ) 

00 


(I) 


<«> 

fa) 


<o) 


Does contractor have a Parts Control Board (PCB) to 
promote proper selection and application of parts used 
in the design ? 

Has contractor established and maintained an up— to- 
date Preferred Parts List (PPL) to be used ay 
designers ? 

Has contractor established derating guidelines for 
derating of electrical /electronic parts electrical 
stresses ? 


Do derating guidelines correspond to specification 
requirements ? 

Has contractor developed part application guidelines 
for proper selection of part types for circuit use? 

Are military grade pans used in the design? 

Are non-standard parts used only when a military 
equivalent part cannot be obtained? 

Where non-standard parts are used do they have 
adequate qualification/ test data and a history of high 
reliability? 

Where non-sta nd a r d parts are used are they pro- 
aired via specifi canon control drawing which speci- 
fies: 

(1) Reliability requirements ? 

(2) Environmental requirements ? 

(3) Test requirements ? 

Has contractor s ubm itted non-standard part data 
for approval per appii cable specification (e.g. . 
MIL-STD- 749/965) ? 

Do parts used in the design meet the environmental 
requirements to which they will be subjected during 
use with respect to: 


(1) 

( 2 ) 

(3) 

<4> 

(5) 


Operating temperature (plus worst case internal 
case temperature rues)? 

Noo-operaung/ storage temper atu re ? 

Humidity? 

Vibration ? 

Shock ? 


H»ve oarta been revie»«d for oroper application, 
have part stresses been calculated ( ) or ( 

and do they meet: 

(1) Derating guidelines? 

(2) Application guidelines 9 

Are established reliability (£R) components and JAN 
semiconductors and microcircuit devices used in the 
design 9 

Where ZR components are used, is the most repre- 
sentative jevei of all ER components used: 


( 1 ) 

( 2 ) 

(2) 

<4) 

(5) 

(6) 


L 

M 

P 

R 

S ' 


Where JAN semiconductors (MIL-S-19S00) are 
used, the most representative level of all sucn de- 
vices used are 

(1) JAN 9 

(2) JaNTX 9 

(3) JaNTXV 9 


Remarks 
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No. 

IP) 


(q) 

<r) 

(•) 

(t) 

(u) 

(V) 

t*> 

(X) 

ty) 

(z) 

(aa) 

(bb) 
(cc ) 
<dd> 
(cc) 

itn 


Item Description 


RELIABILITY (R) DESIGN CHECKLIST 
Yea No 


Where JAN microcircuit* (MIL- M- 38510) or high 
quality microcircuits are used the moet representative 
level of ail such devices used are: 


MDL-M -38510 Class S 
MLL-M-38510 Class B 
MIL -M -38510 Class C 
MEL-STD-883 Qass S 
MIL -STD- 883 Class B 
MIL -STD-883 Class C 
Vendor equivalent to 


( 1 ) 

(2) 

(3) 

(4) 

(5) 

( 6 ) 

( 7 ) 

Do parts meet the interchangeability requirements 
of MIL- STD— 454 Requirement 7? 

Do all parta selected meet the life requirements of 
the equipment? 

Are handling requirement* specified for critical 
and delicate parts susceptible to damage, degradation, 
contamination from shock, vibration, static electric 
discharge, uo cleanliness, etc. ? 

Are assembly and cleaning procedures specified to 
prevent damage to components during assembly on 
PCB’a. chassis, etc. ? 

Have dominant failure modes of a particular part 
type been considered in the selection of that part? 

Are fixed rather than variable components (such as 
resistors, capacitors, inductors, etc.) used in the 
design wherever possible? 

Are all relays, motors, dynamotors, rotary power 
converters, etc. suppressed so as not to produce 
excessive spikes or transients during operation ? 

Are all semiconductor devices silicon rather t*** 
Germanium ? 

Plastic coated and/or encapsulated semiconductor 
devices are not used? 

Do all microcircuits have hermetically sealed ceramic 
cases rather than plaatic cases? 

Do all microcircuits used have at least two potential 
suppliers? 

Do ail unused gates of a digital microcircuit have 
inputs grounded? 

Are the number of expandable gates limited to no 
more than 75% of allowaole number of expendables? 

Where humidity is not controlled are hermetically 
sealed resistors, capacitors, relays, etc., used? 

Are ail power supplies designed and manufactured 
in- nouse ° 

Are parts, even MIL-M- 38510, JANTX, Estaoiished 
Reliaoility ( ER) parts screened at incoming 
inspection: 

(!) 100 %^ 

(2) Sampling plan per * 

♦ 31 Environmentally ' 


Remarks 
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No. 

24 


(*) 

(b) 


(C) 


(d) 

fe) 


<n 

<g> 

(b) 


RELIABILITY <R) DESIGN CHECKLIST 


Item Poser lotion 

Developmental Test Program 

Is contractor conducting a developmental test pro* 
gram? 

Does developmental test program include: 

(1) All critical assembling ? 

(2) Each assembly with a unique form factor ? 

(3) Critical non-standard parts? 

Does developmental testing Include environmental 
testing at or above the levels specified for qualifica- 
tion: 

(1) High and low temperature? 

(2) Vibration? 

(3) Shock? 

(4) Humidity? 

Are performance requirements checked over re- 
quired operating temperature levels? 

Are life testa or reliability tssu of critical com- 
ponents /subassemblies being or have they been 
conducted? 

Is "Step Stress" testing being performed on sub- 
assemblies, etc. . to determine design margins? 

Is developmental test program monitored by the 
reliability group or does the reliability group provide 
inputs to developmental testing? 

Are failure data and maintenance data collected 
during developmental testing for determining need 
for reliability improvement? 


Yes No 


Remarks 


7-C-24* 



MIL-HDBK-338 
15 OCTOBER 1984 



RELIABILITY (R) DESIGN CHECKLIST 

No. 

Item Descnocion 

Yes 

No 

25 

Reliability Anaivses 



U) 

Have the following reliability analyses been per* 
formed: 

(1) Reliability Mathematical Models? 




(21 Reliability Apportionments? 




(3) Reliability Predictions? 




(41 Failure Modes and Effects Analyses? 




(5) Criticality Analyses? 




(6) Circuit Analysis (nominal and worst cases)? 




(?) Thermal Analysis? 




(8) Sneak Circuit Analysis? 



(b) 

Do predictions meet apportioned values ? 

mmmm 


(c) 

Do predictions meet numerical reliability speci- 
fication requirements? 



<d) 

Have the results o / the predictions been used to 
increase equipment reliability by: 

(1) Reduction of circuit complexity? 




(2) Reduction of ambient temperature conditions ? 

(3) Reduction of internal temperature rises? 

— 

— 


(4) Reduction of part stresses by further derating? 




(5) Increase of part quality levels ? 




(6) Addition of redundancy? 



(e) 

Has a numerical approach for Criticality Analysts 
been used? 



(f) 

Does the numerical criticality analysis consider: 
(1) Frequency of failure? 




(2) Degree of effect on system performance? 




(3) Difficulty to diagnose and/or repair? 




(4) Personnel or equipment safety? 

- 


U) 

Have all critical inodes of system failure been 
identified ? 



<*> 

Have critical items been ranked as to criticality? 

— 



(k) 

Has the use of limited life items been kept to a 
minimum ? 



(I) 

Have the analyses considered the effects of 




storage, transportation and handling on failure 
modes, effects and failure rates? 



<m) 

Has the use of circuit analysis provided a stable, 
design over the worst case conditions? 



(n) 

Has protective circuitry been utilized in the 
equipment oesign ? 
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No. 

26 


Item Description 
Bum-ia Program 


REUABIUTY (R) DESIGN CHECKLIST 
Yea No 


<•> 


(b) 


(c) 

Id) 

(•> 

<*) 

<*) 


Doea the contractor Impose bum-in ax- 
il) Component level ? 

(2) Subaaeembly/module level ? 

(3) Equipment/ ay stem level ? 

Ia burn- la performed under; 

<1) Temperature (elevated) ? 

(2) Temperature cycling? 

(3) Vibration? 

Are lengths of bum-in adequate for each level? 

Do spares receive same burn-in aa modules/ 
subassembly level? 

Do ail equipments/systems receive the same 
amount of bum-in? 

Dow coo motor have a UUure free bora-la re- 
quirement prior to acceptance of the equipment? 

Is random vibration performed ? 

(1) Equipment level? 

(2) *'r* level ? 

(3) Frequency range ? “ 

(4) Time duration? 



Remarks 
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No. 


27 


(*> 

fb> 


(Cj 

(d) 

<«) 

(*) 

(f) 


IS) 


(hi 

ft) 

(j) 

fk) 

(l) 

(m) 
<n) 


Item Description 


RELIABILITY <R) DESIGN CHECKLIST 
Yet No 


Failure Reporting Analysts and Corrective Action 
fFRACAl Program 


H At coo tractor implemented a FRACA program ? 

Doea FRACA program cover failures during: 

(11 Source inspection at subcontractor* < plant? 

(2) Incoming inspection? 

(3) In-proceti inspection? 

(4) Development teats ? 

(5) Subassembly /module teat? 

(6) Equipment integration and checkout? 

(?) Equipment burn-in? 

(8) Equipment formal teatt: 

(a) Acceptance teats? 

fb) Environmental /qualification tests? 

ic) Reliability /Maintainability teats ? 

Does contractor have in-house facilities for per- 
forming detailed failure analysis? 

Is failure analysis condu c ted for all failures? 

Are failures summarized by pert number and failure 
type to determine trends and patterns? 

Has contractor established thresholds (percent defec- 
tive or failure rate) for determining need for correc- 
tive action ? 

Does failure report form contain the necessary in- 
formation with regards to: 

(1) Identification of failed part subassembly, 
assembly, etc. ? 

(2) Elapsed time meters (for failure at equipment 
level) ? 

(3) Failure symptoms? 

(4) Effect of failure on system/equipment? 

(3) Test and environmental conditions at time of 
failure ? 

(6) Suspected cause of failure? 

Is tbe same type of FRaCA program imposed upon 
sudc on tractors of critical subassemblies ? 

Are subcontractor failure reports included in coo- 
tractor failure summaries ? 

Are ail failure reports, analyses and corrective 
actions reviewed by the reliability group? 

Are failure treads monitored by the reliability 
group 0 

Are corrective actions involving design changes 
tested in the equipment for an adequate period of 
tune prior to their formalization? 


Are corrective action investigations reopened upon 
a recurrence of the same type of failure? 

Are proposed corrective actions referred to the 
^ocurTng Activity 'or concurrenct? 


Remar 
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No. 

28 


RELIABILITY (R) design checklist 


Item Description 

Reliability Demonstration Test Planning 


Yes No 


U> 

(b) 

(c) 

<<i) 

(•) 

m 

U) 

(b) 

u> 

<J) 

(k) 

(l) 
(m) 

in) 

(o) 

<P> 


wui teat simulate o periling profUe that wUl be sees 
aboard ship? 

Will all modes o i equipment operation be tested? 

Is definition of failure in accordance with contract 
specification requirements ? 

Are relevant and oon-relevant failure definitions 
adequately defined? 

Will test be performed under envir onmental levels 
specified by the contract specifications ? 

Will fcwrn-in to be performed on reliabUicy test units 
be no more or no less than that specified for pro- 
duction units? 

Non-operating: and equipment standby time will be 
discounted from applicable test time for validating 
reliability , true ? 

No Preventive Maintenance other than that contained 
in technical manuals and approved by the Navy will 
be performed during the test, true ? 

Performance checks capable of checking the complete 
equipment failure rate, performed no less frequently 
than daily have been defined for the test, true? 

Test will be performed per agreed schedule, true? 

Procuring Activity will be notified of the exact 
test date at least 30 days prior to the test, true? 

All interfaces are simulated or stimulated ? 

All interfaces are real? 

If interfaces are real, is GFE required? 

If G Ft is required, has a request been 
to obtain GFE? 

Is test DD 1423 documentation on schedule ? 


Remarks 
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1 Is design simple? Minimum number of parts? 

2 . It it designed into a unified overall system rather than as an accumulation of parts, etc.? 

3. Is the item compatible with system in which it is used? 

4. 1$ the item properly integrated and installed in the system? 

5. Are there adequate indicators to verify critical functions? 

6. Has reliability for spares and repair parts been considered? 

7. Are reliability requirements established for critical items? For each part? 

8. Is there specific reliability design criteria for each item? 

9* Have reliability tests been established? 

10. Are standard high-reliability parts being used? 

11. Are unreliable parts identified? 

12. Has the failure rate for each part or part class been established? 

13. Have parts been selected to meet reliability requirements? 

14. Have below-state-of-the-art parts or problems been identified? 

15. Has shelf life of parts been determined? 

1 6. Have limited-life parts been identified, and inspection, and replacement requirements specified? 

17. Have criticaJ parts which required special procurement, testing, and handling been identified? 

18. Have stress analyses been accomplished? 

19. Have derating factors been used in the application of parts? 

20. Have safety factors and safety margin been used in the application of parts? 

21. Are circuit safety margins ample? 

22 . Have standard and proven circuits been utilized? 

23. Has the need for the selection of parts (matching) been eliminated? 

24. Have circuit studies been made considering variability and degradation of electrical parameters 
of parts? 

25. Have solid-state devices been used where practicable? ____ ___ 

FIGURE 7. LI. 4-2: TYPICAL QUEST IONS CHECKLIST FOR THE DESIGN REVIEW (SHEET l_ o£ 
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26 . 


27 . 

28 . 


29 . 


30 . 

31 . 


32 . 


33 . 


34 . 

35 . 


1$ the reliability or MTBF of the item based on actual application of the parts? 

a. Comparison made with reliability goal? 

b. Provision for necessary design adjustments? 

Are the best available methods for reducing the adverse effects of operational environments on 
critical parts being utilized? 

Has provision been made for the use of electronic failure prediction techniques, including marginal 
testing? 

Is there provision for improvements to eliminate design inadquacies observed in tests? 

Have normal modes of failure and the magnitude of each mode for each item or critical part been 
identified? 

In the application of failure rates of items to reliability equations, have the following effects been 
considered? 

a. External effects on the next higher level which the item is located. 

b. Internal effects on the item. 

c. Common effects, or direct effect of one item on another item, because of mechanical 
or electro-mechanical linkage. 

Has redundancy been provided where needed to meet specified reliability? 

Has failure mode and effects analyses been adequately covered by design? 

Have the risks associated with critical item failures been identified? Accepted? Has design action 
been taken? 

Does the design account for early failure, useful life and wear-out? 



IE 7.11.4-2: TYPICAL QUESTIONS CHECKLIST FOR THE DESIGN REVIEW (SHEET 2 of 2 ) 
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